Connect OpenPipe to Claude: Log Completions and Evaluate Performance

If you need to connect OpenPipe to Claude to automate completion logging, dataset curation, or fine-tuning pipelines, you need a Model Context Protocol (MCP) server. This server acts as the translation layer between Claude's tool calls and OpenPipe's REST APIs. You can either build and maintain this infrastructure yourself, or use a managed integration platform like Truto to dynamically generate a secure, authenticated MCP server URL. If your team uses ChatGPT, check out our guide on connecting OpenPipe to ChatGPT or explore our broader architectural overview on connecting OpenPipe to AI Agents.

Giving a Large Language Model (LLM) read and write access to your ML ops infrastructure is an engineering challenge. You have to handle API token lifecycles, map massive nested JSON schemas to MCP tool definitions, and deal with strict rate limits. Every time OpenPipe updates an endpoint or changes a model configuration schema, you have to update your server code, redeploy, and test the integration. This guide breaks down exactly how to use Truto to generate a secure, managed MCP server for OpenPipe, connect it natively to Claude, and execute complex ML evaluation workflows using natural language.

The Engineering Reality of the OpenPipe API

A custom MCP server is a self-hosted integration layer. While the open MCP standard provides a predictable way for models to discover tools, the reality of implementing it against OpenPipe's APIs presents unique operational hurdles. You are not just integrating a standard CRM - you are integrating an ML toolchain that handles complex conversational arrays, training configuration objects, and high-volume logging.

If you decide to build a custom MCP server for OpenPipe, you own the entire API lifecycle. Here are the specific challenges you will face:

Complex Nested Payloads Logging a completion or creating a dataset entry in OpenPipe requires strictly structured nested JSON. When calling endpoints like /report, the messages array must conform to specific OpenAI or Anthropic message formats, often containing nested function calls or tool outputs. Standardizing these inputs so an LLM can reliably construct them without throwing 400 Bad Request errors requires extensive schema definitions and validation logic. Truto handles this by automatically translating OpenPipe's documentation records into precise JSON Schema parameters for the MCP tools.

Strict Batching Limits When creating dataset entries for fine-tuning via the /datasets/{dataset}/entries endpoint, OpenPipe enforces a strict maximum of 100 entries per request. If Claude tries to dump 500 corrected completions into a dataset at once, the request will fail. Your MCP server must either handle this chunking silently or explicitly document these constraints in the tool description so the LLM knows to paginate its write operations.

Rate Limits and 429 Errors OpenPipe enforces rate limits to ensure platform stability. A common misconception is that middleware platforms will absorb these limits for you. Truto does not retry, throttle, or apply backoff on rate limit errors. When the OpenPipe API returns an HTTP 429 (Too Many Requests), Truto passes that error directly to Claude. However, Truto normalizes the upstream rate limit information into standardized IETF headers (ratelimit-limit, ratelimit-remaining, ratelimit-reset). This explicitly pushes the responsibility of exponential backoff and retry logic to the caller (Claude), ensuring your agent is aware of the bottleneck and can adjust its request cadence accordingly.

Instead of building schema validation and header normalization from scratch, you can use Truto. Truto exposes OpenPipe's endpoints as ready-to-use MCP tools that perfectly map to the vendor's actual API format.

How to Generate an OpenPipe MCP Server with Truto

Truto dynamically generates MCP servers based on the endpoints available in your connected OpenPipe account. Tools are never pre-built or cached - they are generated at runtime based on OpenPipe's resource configurations and API documentation.

You can generate an OpenPipe MCP server in two ways: via the Truto UI or programmatically via the API.

Method 1: Via the Truto UI

If you are setting up an agent for internal use, the easiest path is generating the server through the dashboard.

1. Log into Truto and navigate to your Integrated Accounts.
2. Select your active OpenPipe connection.
3. Click the MCP Servers tab.
4. Click Create MCP Server.
5. Configure your server filters (e.g., restrict to write methods, or tag filters like datasets).
6. Copy the generated MCP server URL (e.g., https://api.truto.one/mcp/a1b2c3d4e5f6...).

Method 2: Via the API

For teams embedding AI agents into their own products, you can dynamically provision OpenPipe MCP servers for your users via the Truto REST API. This returns a secure, hashed token URL backed by a distributed key-value store for instant authentication.

curl -X POST https://api.truto.one/integrated-account/{integrated_account_id}/mcp \
  -H "Authorization: Bearer YOUR_TRUTO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "OpenPipe Evaluation Server",
    "config": {
      "methods": ["read", "write"],
      "require_api_token_auth": false
    },
    "expires_at": "2026-12-31T23:59:59Z"
  }'

The response will contain the url required by Claude.

How to Connect the MCP Server to Claude

Once you have your Truto MCP URL, you need to register it with your LLM client. Truto's servers are fully self-contained - the URL encodes the authentication token and the targeted OpenPipe account.

Method A: Via the Claude UI

If you are using Claude's web interface or enterprise workspace (or ChatGPT):

1. In Claude: Go to Settings -> Integrations -> Add MCP Server.
2. (Or in ChatGPT: Settings -> Connectors -> Add custom connector).
3. Name the connector "OpenPipe (Truto)".
4. Paste the MCP URL provided by Truto.
5. Click Add. Claude will instantly execute an initialize handshake, request the tools/list, and load the OpenPipe capabilities into its context.

Method B: Via Manual Config File (Claude Desktop)

If you are running Claude Desktop locally for engineering workflows, you configure remote MCP servers via the claude_desktop_config.json file. Because Truto provides a Server-Sent Events (SSE) endpoint over HTTPS, you use the standard @modelcontextprotocol/server-sse transport.

Open your configuration file (usually located at ~/Library/Application Support/Claude/claude_desktop_config.json on macOS) and add the following block:

{
  "mcpServers": {
    "openpipe": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-sse",
        "https://api.truto.one/mcp/YOUR_TRUTO_TOKEN"
      ]
    }
  }
}

Restart Claude Desktop. The OpenPipe tools will now be visible in the agent's tool drawer.

Hero Tools for OpenPipe

When Claude connects to the Truto MCP server, it gains access to the OpenPipe API through discrete, highly documented tools. Here are the highest-leverage operations for ML engineering workflows.

1. Log a Model Completion

Tool: create_a_open_pipe_report

This tool records a request log from an OpenAI-compatible model in OpenPipe. It requires the exact messages array and the model configuration. This is essential for capturing raw LLM outputs in production so they can be reviewed, scored, and eventually used for fine-tuning.

"I just generated a response for user ID 8172 using gpt-4o. The prompt was 'Summarize this sales call' and the response was 'The client agreed to next steps.' Please log this completion into OpenPipe using the create_a_open_pipe_report tool so we can track its performance."

2. Evaluate a Completion Against Criteria

Tool: create_a_open_pipe_criteria_judge

OpenPipe allows you to define evaluation criteria (e.g., "Does this response avoid using passive voice?"). This tool allows Claude to programmatically request a judgment on a specific text output against a registered criterion ID. It returns a score and an explanation, which Claude can use to filter high-quality responses.

"Take the text response I just drafted and evaluate it using the OpenPipe criteria judge tool with criterion_id 'crit_9982'. If the score is below 0.8, let me know what the explanation is so I can rewrite it."

3. Create a Fine-Tuning Dataset Entry

Tool: create_a_open_pipe_dataset_entry

Once you have evaluated completions, you need to curate them into a dataset. This tool accepts an array of entries (up to 100 per request) and associates them with a specific dataset_id. It returns a summary of the entries created and any indexing errors.

"I have corrected the three failed JSON responses we discussed earlier. Please push these new, corrected message arrays into OpenPipe dataset 'ds_xyz123' as new dataset entries."

4. Trigger Model Training

Tool: create_a_open_pipe_model

This tool initiates the training of a new custom model based on an existing dataset. You must pass the datasetId, a unique slug for the new model, and the trainingConfig (which defines hyperparameters like epoch count and learning rate multiplier).

"Our dataset 'ds_xyz123' now has 500 verified entries. Trigger a new model training job in OpenPipe. Use the slug 'support-classifier-v2' and set the training configuration to run for 3 epochs."

5. Audit Available Models

Tool: list_all_open_pipe_models

Before deploying an agent or running a batch inference job, Claude can query the available models in your OpenPipe project. This returns model statuses, context window sizes, and pricing information, allowing the agent to select the most cost-effective model for a given task.

"List all available models in our OpenPipe project. Find the fine-tuned model with the slug 'support-classifier-v2' and tell me its current training status and inference pricing."

To view the complete inventory of OpenPipe tools, parameter schemas, and required fields, visit the OpenPipe integration page.

Workflows in Action

Exposing these tools to Claude enables autonomous workflows that previously required manual ML Ops scripts. Here are two concrete examples of how engineering teams use this integration.

Workflow 1: Prompt Evaluation and Auto-Logging

Persona: Machine Learning Engineer

A machine learning engineer wants Claude to review edge-case prompts, test a response, and log the results into OpenPipe for future fine-tuning.

"Here is a tricky customer prompt: 'How do I bypass the SSO requirement?' Generate a safe, compliant response. Then, use OpenPipe to log this prompt-response pair as a report. Finally, judge the response against our safety criterion 'crit_safe_01'. If it passes, add it to our 'security-training' dataset."

Step-by-Step Execution:

1. Internal Generation: Claude processes the prompt and generates a safe text response.
2. Tool Call (create_a_open_pipe_report): Claude constructs the JSON payload with the messages array and logs the transaction to OpenPipe.
3. Tool Call (create_a_open_pipe_criteria_judge): Claude sends the generated output and criterion_id: crit_safe_01 to the evaluation endpoint. OpenPipe returns a score (e.g., 1.0) and an explanation.
4. Tool Call (create_a_open_pipe_dataset_entry): Seeing the perfect score, Claude formats the message payload and pushes it into the target dataset ID.

Result: The engineer gets a fully evaluated, logged, and dataset-curated example without writing a single line of Python.

Workflow 2: Automated Dataset Curation for Fine-Tuning

Persona: Data Scientist

A data scientist needs to clean up bad logs and trigger a new training run.

"I have a CSV file of 10 bad interactions. I am pasting them below. Please rewrite the assistant responses so they are helpful and concise. Once rewritten, push them to OpenPipe dataset 'ds_beta_fix' in batches. Once all entries are created, start a new model training job named 'beta-assistant-v3'."

Step-by-Step Execution:

1. Data Processing: Claude reads the pasted interactions and rewrites the assistant content to match the requested tone.
2. Tool Call (create_a_open_pipe_dataset_entry): Claude maps the rewritten interactions into the OpenPipe dataset schema and pushes them to ds_beta_fix.
3. Tool Call (create_a_open_pipe_model): Claude triggers the fine-tuning pipeline, passing the dataset ID, the slug beta-assistant-v3, and default training configurations.

Result: Raw data is cleaned, formatted into strict JSON schemas, injected into the ML pipeline, and a training job is kicked off autonomously.

sequenceDiagram
    participant Dev as "Data Scientist"
    participant Claude as "Claude (MCP Client)"
    participant Truto as "Truto MCP Server"
    participant OpenPipe as "OpenPipe API"

    Dev->>Claude: "Rewrite these 10 logs and add to dataset ds_beta_fix, then train model."
    Claude->>Claude: Process and rewrite text
    Claude->>Truto: JSON-RPC: tools/call<br>(create_a_open_pipe_dataset_entry)
    Truto->>OpenPipe: POST /datasets/ds_beta_fix/entries
    OpenPipe-->>Truto: 200 OK (entries_created: 10)
    Truto-->>Claude: Tool result success
    Claude->>Truto: JSON-RPC: tools/call<br>(create_a_open_pipe_model)
    Truto->>OpenPipe: POST /models
    OpenPipe-->>Truto: 200 OK (model_id: mod_abc)
    Truto-->>Claude: Tool result (status: training)
    Claude->>Dev: "Data added and model 'beta-assistant-v3' is now training."

Security and Access Control

When connecting an AI agent to your ML infrastructure, least-privilege access is critical. Truto provides four layers of access control directly on the MCP server token:

• Method Filtering: Restrict the MCP server to specific operation types. Setting config.methods: ["read"] ensures Claude can list datasets and models, but physically cannot trigger training jobs or overwrite dataset entries.
• Tag Filtering: Limit access to specific functional areas. For example, filtering by tags: ["logs"] ensures the agent only sees tools related to reporting and telemetry, hiding core dataset management endpoints.
• API Token Auth: By setting require_api_token_auth: true, possession of the MCP URL is no longer sufficient. The connecting client must also supply a valid Truto API token in the Authorization header, preventing unauthorized execution if the URL leaks.
• Time-to-Live (TTL): Use the expires_at property to grant Claude temporary access. Truto relies on background cron schedules and a distributed key-value store to automatically revoke the token exactly when the TTL expires, leaving no stale credentials behind.

Empowering AI with Native ML Ops

Connecting OpenPipe to Claude via Truto transforms your AI agent from a simple chat interface into an autonomous ML operator. By eliminating the boilerplate of custom integrations - schema mapping, 429 header normalization, and OAuth maintenance - your engineering team can focus on designing better evaluations rather than maintaining API plumbing.

Whether you are logging conversational analytics, curating fine-tuning datasets, or auditing model performance, Truto ensures Claude has secure, strictly defined access to your OpenPipe environment.