Connect OpenPipe to ChatGPT: Train Custom Models and Manage Datasets

You need to connect OpenPipe to ChatGPT to automate dataset curation, trigger custom model training jobs, and run LLM evaluations natively from your chat interface. If your team relies on other AI interfaces, check out our guides on connecting OpenPipe to Claude and connecting OpenPipe to AI Agents.

Giving a Large Language Model (LLM) read and write access to your fine-tuning pipeline is an engineering challenge. You either spend weeks building, hosting, and maintaining a custom Model Context Protocol (MCP) server, or you use a managed infrastructure layer that handles the boilerplate. This guide breaks down exactly how to use Truto to generate a secure, managed MCP server for OpenPipe, connect it natively to ChatGPT, and execute complex model-training workflows using natural language.

The Engineering Reality of the OpenPipe API

A custom MCP server is a self-hosted integration layer that translates an LLM's tool calls into REST API requests. While Anthropic's open standard provides a predictable way for models to discover tools, the reality of implementing it against specific vendor APIs requires deep domain knowledge.

If you decide to build a custom MCP server for OpenPipe, you are responsible for the entire API lifecycle. You are not just building basic CRUD operations - you have to account for the specific architectural quirks of a data-intensive fine-tuning platform.

Dataset Batching Constraints LLMs are notoriously bad at pacing themselves. When an agent attempts to bulk-upload thousands of synthetic data rows into an OpenPipe dataset, it will try to send everything in a single massive payload. The OpenPipe API strictly caps dataset entry creation at 100 entries per request. Your MCP server must intercept these requests, enforce the 100-entry limit within the JSON schema exposed to the LLM, and explicitly instruct the model on how to chunk its data uploads. If you fail to do this, the API rejects the payload, and the LLM hallucinates a successful upload.

Asynchronous Training State Management Triggering a model training job via the OpenPipe API is instantaneous, but the training itself takes time. When an LLM calls the endpoint to create a model, it receives a model object with a pending status. The LLM needs a secondary polling mechanism to check the training status. If your MCP server does not expose a separate tool for fetching specific model states by ID and instruct the LLM on polling intervals, the agent will assume the model is ready for inference immediately.

Criteria Evaluation Idiosyncrasies OpenPipe's criteria judging endpoints require exact string matching against pre-configured criterion_id values. When an LLM evaluates a prompt completion, it cannot pass arbitrary strings. Your MCP server must dynamically map the available criteria IDs from the OpenPipe project into the query schema so the LLM knows exactly which identifiers are valid.

Architecture: The Managed MCP Approach

Instead of forcing your engineering team to build custom API translation logic, manage token lifecycles, and handle error normalization, Truto acts as the execution layer.

The system derives MCP tools dynamically from OpenPipe's API definitions. Tools are not hardcoded. When ChatGPT requests the list of available tools, Truto maps the raw OpenPipe REST endpoints into JSON-RPC 2.0 compatible schemas in real-time.

When the LLM executes a tool call, Truto translates the JSON-RPC payload into standard HTTP requests, injects the correct authentication credentials for the integrated OpenPipe account, and passes the response back to the LLM.

sequenceDiagram
    participant ChatGPT as ChatGPT
    participant TrutoMCP as Truto MCP Server
    participant OpenPipe as OpenPipe API

    ChatGPT->>TrutoMCP: POST /mcp/:token (tools/call: create_a_open_pipe_dataset)
    TrutoMCP->>TrutoMCP: Validate Token & Parse Schema
    TrutoMCP->>OpenPipe: POST /v1/datasets (Bearer Token)
    OpenPipe-->>TrutoMCP: HTTP 200 OK { id, name, created }
    TrutoMCP-->>ChatGPT: JSON-RPC Result

Handling Rate Limits

OpenPipe enforces rate limits to prevent API abuse. It is a critical architectural decision that Truto does not retry, throttle, or apply backoff on rate limit errors.

When the OpenPipe API returns an HTTP 429 Too Many Requests, Truto passes that error directly to the caller. Truto normalizes the upstream rate limit information into standardized headers (ratelimit-limit, ratelimit-remaining, ratelimit-reset) per the IETF specification. The caller (the LLM client or agent framework) is fully responsible for implementing its own retry and exponential backoff logic. Truto does not absorb rate limit errors.

Step 1: Generating the OpenPipe MCP Server

To connect ChatGPT to OpenPipe, you first need a secure MCP server URL. Truto ties this URL to a specific integrated OpenPipe account in your tenant.

You can generate this server via the Truto UI or programmatically via the API.

Method A: Via the Truto UI

1. Navigate to the Integrated Accounts page in the Truto dashboard.
2. Select your connected OpenPipe account.
3. Click the MCP Servers tab.
4. Click Create MCP Server.
5. Select your desired configuration (e.g., restrict to read-only methods or filter by specific tags).
6. Copy the generated MCP server URL. You will paste this directly into ChatGPT.

Method B: Via the API

For platform teams building automated provisioning workflows, you can generate the MCP server programmatically using your Truto API key.

curl -X POST https://api.truto.one/integrated-account/{integrated_account_id}/mcp \
  -H "Authorization: Bearer YOUR_TRUTO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "OpenPipe Dataset Admin",
    "config": {
      "methods": ["read", "write"],
      "require_api_token_auth": false
    },
    "expires_at": null
  }'

The API returns a fully qualified JSON-RPC endpoint containing a cryptographic hash linked to the specific account.

{
  "id": "mcp_abc123",
  "name": "OpenPipe Dataset Admin",
  "url": "https://api.truto.one/mcp/a1b2c3d4e5f6..."
}

Step 2: Connecting the MCP Server to ChatGPT

Once you have your Truto MCP server URL, you must register it with your ChatGPT environment. You can do this through the ChatGPT user interface or via manual configuration for custom agent deployments.

Method A: Via the ChatGPT UI

OpenAI provides native support for remote MCP connectors on Pro, Team, and Enterprise plans.

1. Open ChatGPT and navigate to Settings -> Apps -> Advanced settings.
2. Enable the Developer mode toggle (MCP support requires this flag).
3. Under the MCP servers / Custom connectors section, click Add new server.
4. Enter a name (e.g., "OpenPipe (Truto)").
5. Paste the Truto MCP URL into the Server URL field.
6. Save the configuration.

ChatGPT will immediately ping the initialization endpoint, negotiate capabilities, and fetch the list of OpenPipe tools.

Method B: Via Manual Config File (SSE Transport)

If you are running a custom ChatGPT-compatible client, a headless agent, or a local developer environment, you can configure the MCP connection manually. While Claude uses a standard claude_desktop_config.json, headless agent frameworks often require a Server-Sent Events (SSE) transport wrapper for remote servers.

You can use the official @modelcontextprotocol/server-sse package to proxy the connection to Truto.

{
  "mcpServers": {
    "openpipe-truto": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-sse",
        "--url",
        "https://api.truto.one/mcp/a1b2c3d4e5f6..."
      ]
    }
  }
}

Security and Access Control

Exposing an infrastructure-level tool like OpenPipe to an LLM requires strict governance. Truto MCP servers include built-in security controls that execute at the routing layer:

• Method Filtering: Use config.methods to strictly limit the server to read, write, or specific operations (e.g., allowing ["get", "list"] while blocking create or delete).
• Tag Filtering: Scope the available tools to specific integration domains using config.tags (e.g., restricting access to only datasets while hiding models).
• Time-to-Live (TTL): Set an expires_at timestamp. Once the timestamp passes, the server URL is automatically invalidated and the connection is severed.
• Secondary Authentication: Enable require_api_token_auth to force the client to pass a valid Truto API token in the Authorization header. This ensures that even if the MCP URL is leaked, unauthorized users cannot execute tools.

OpenPipe Hero Tools

Once connected, ChatGPT gains access to the OpenPipe Proxy API tools. Truto derives tool names automatically from the integration schemas. Here are the highest-leverage tools available for OpenPipe workflows.

Create a Dataset

Creates a new fine-tuning dataset container within your OpenPipe project.

Tool name: create_a_open_pipe_dataset Context: Use this to initialize a new dataset before pushing synthetic data. It returns the id required for subsequent entry uploads.

"Create a new OpenPipe dataset named 'Customer_Support_Triage_v2'. Return the exact dataset ID so we can use it to upload log entries later."

Create Dataset Entries

Uploads new data rows to a specific dataset.

Tool name: create_a_open_pipe_dataset_entry Context: Requires the dataset_id and an array of entries. Note the strict API limit: a maximum of 100 entries per request. If you have more data, the LLM must iterate over the list and call the tool multiple times.

"Take these 50 synthetic customer service JSON objects and upload them as entries to the dataset ID 'ds_987654321'. Ensure the role formatting matches the OpenPipe schema."

Train a Custom Model

Kicks off a fine-tuning job on a specific dataset using a defined base model.

Tool name: create_a_open_pipe_model Context: Requires the datasetId, a slug for the new model name, and the trainingConfig parameters. The endpoint returns a model object with a pending status.

"Start a fine-tuning job using dataset ID 'ds_987654321'. Name the model 'support-triage-llama-3'. Use standard training configuration parameters."

Check Model Training Status

Fetches the current status of an OpenPipe model by its ID.

Tool name: get_single_open_pipe_model_by_id Context: Training is asynchronous. Use this tool to poll the API and check if the model is ready, failed, or still processing.

"Check the current status of the OpenPipe model with ID 'mod_123456'. Tell me if the fine-tune is complete and what the final context window size is."

Judge Criteria

Evaluates a prompt output against a predefined OpenPipe criterion.

Tool name: create_a_open_pipe_criteria_judge Context: Highly useful for LLM-as-a-judge workflows. Requires a valid criterion_id and the output to evaluate. Returns a score and explanation.

"Evaluate this response string against OpenPipe criterion ID 'crit_abc987'. Return the numerical score and the explanation for why the model received that score."

Update Log Metadata

Updates tags or metadata for logged OpenPipe calls based on specific filters.

Tool name: create_a_open_pipe_logs_update_metadata Context: Used to retroactively label production logs. For example, if a user flags a bad response, you can update the log metadata to exclude it from future fine-tuning datasets.

"Find the log entry with request ID 'req_555' and update its metadata to include 'quality_flag: poor' and 'needs_review: true'."

For a complete list of available tools, query schemas, and response formats, view the OpenPipe integration page.

Workflows in Action

Connecting OpenPipe to ChatGPT enables complex, multi-step engineering tasks to be executed conversationally. Here is how specific personas use this setup in production.

Scenario 1: The Automated Evaluation Pipeline

Machine learning engineers need to evaluate prompt completions and tag poor responses so they are not included in future training runs.

"I need to evaluate the completion output for request ID 'req_999'. Run it against the 'Hallucination Check' criterion (ID: crit_444). If the score is less than 0.8, update the metadata on that log to flag it as 'exclude_from_training'."

Execution flow:

1. ChatGPT calls create_a_open_pipe_criteria_judge passing criterion_id: crit_444 and the text output of the target request.
2. OpenPipe evaluates the text and returns a JSON payload containing score: 0.5 and explanation: "The model invented a feature that does not exist."
3. ChatGPT parses the score. Because it is below the 0.8 threshold, the LLM determines it must execute the next step.
4. ChatGPT calls create_a_open_pipe_logs_update_metadata with the filter request_id: req_999 and updates the metadata with the exclusion tags.

Result: The engineer successfully runs an LLM-as-a-judge evaluation and updates production logs without writing a single script or opening the OpenPipe dashboard.

Scenario 2: End-to-End Dataset Creation and Training

AI product managers frequently generate synthetic data to test new fine-tunes.

"I have 40 synthetic Q&A pairs for our new API docs. Create a new OpenPipe dataset called 'API_Docs_Synthetic'. Once created, upload these 40 pairs as entries. Finally, trigger a new fine-tuning job on this dataset and let me know the model ID to track."

Execution flow:

1. ChatGPT calls create_a_open_pipe_dataset with the name "API_Docs_Synthetic".
2. OpenPipe returns the new dataset object containing id: ds_777.
3. ChatGPT maps the 40 Q&A pairs into the required JSON array format and calls create_a_open_pipe_dataset_entry using the new ds_777 ID.
4. Upon receiving the success response for the entry upload, ChatGPT calls create_a_open_pipe_model passing datasetId: ds_777 and the training configuration.
5. OpenPipe returns the pending model object containing id: mod_888.

Result: The user defines the data and intent, and ChatGPT orchestrates the exact sequence of API calls to build the dataset and queue the training job, returning the tracking ID to the user.

Moving Fast Without Breaking the API

Connecting ChatGPT to OpenPipe via MCP shifts the integration paradigm from rigid, hardcoded scripts to dynamic, agentic workflows. By utilizing Truto as the managed MCP layer, your engineering team bypasses the complexities of custom schema generation, authentication management, and protocol handling.

You retain complete control over the OpenPipe API surface via method filtering and strict schemas, ensuring the LLM operates securely within its bounds. The result is a highly capable AI assistant that can manage your datasets, evaluate model performance, and trigger fine-tuning jobs directly from the chat window.