---
title: "Connect OpenPipe to AI Agents: Automate Dataset Curation and Fine-Tuning"
slug: connect-openpipe-to-ai-agents-automate-dataset-curation-and-fine-tuning
date: 2026-07-08
author: Sidharth Verma
categories: ["AI & Agents"]
excerpt: "Learn how to connect OpenPipe to AI agents using Truto's /tools endpoint. Automate dataset curation, evaluate completions, and manage fine-tuning workflows."
tldr: "Connect OpenPipe to AI agents natively using Truto. This guide covers how to bypass legacy API quirks, fetch dynamic tools, handle batch limits, and automate fine-tuning pipelines."
canonical: https://truto.one/blog/connect-openpipe-to-ai-agents-automate-dataset-curation-and-fine-tuning/
---

# Connect OpenPipe to AI Agents: Automate Dataset Curation and Fine-Tuning


You want to connect OpenPipe to an AI agent so your internal systems can independently [evaluate completions](https://truto.one/connect-openpipe-to-claude-log-completions-and-evaluate-performance/), curate training datasets, execute fine-tuning jobs, and [manage custom models](https://truto.one/connect-openpipe-to-chatgpt-train-custom-models-and-manage-datasets/) based on historical application logs. Here is exactly how to do it using Truto's `/tools` endpoint and SDK, bypassing the need to manually code API wrappers or maintain complex polling logic for asynchronous training jobs.

Giving a Large Language Model (LLM) read and write access to your OpenPipe instance is an engineering headache. You either spend weeks building, hosting, and maintaining a custom connector that understands the nuances of dataset batching and legacy endpoint deprecations, or you use a managed infrastructure layer that handles the boilerplate for you. If your team uses ChatGPT, check out our guide on [connecting OpenPipe to ChatGPT](https://truto.one/connect-openpipe-to-chatgpt-train-custom-models-and-manage-datasets/), or if you are building on Anthropic's models, read our guide on [connecting OpenPipe to Claude](https://truto.one/connect-openpipe-to-claude-log-completions-and-evaluate-performance/). For developers building custom autonomous workflows, you need a programmatic way to fetch these tools and bind them to your agent framework.

This guide breaks down exactly how to fetch AI-ready tools for OpenPipe, bind them natively to an LLM using LangChain (or any framework like LangGraph, CrewAI, or Vercel AI SDK), and execute complex dataset curation and fine-tuning workflows. For a deeper look at the architecture behind this approach, refer to our research on [architecting AI agents and the SaaS integration bottleneck](https://truto.one/architecting-ai-agents-langgraph-langchain-and-the-saas-integration-bottleneck/).

## The Engineering Reality of Custom OpenPipe Connectors

Building AI agents is easy. Connecting them to external SaaS APIs is hard. Giving an LLM access to external data sounds simple in a prototype. You write a Node.js function that makes a fetch request and wrap it in an `@tool` decorator. In production, this approach collapses entirely, especially with a specialized machine learning operations platform like OpenPipe.

If you decide to build a custom OpenPipe integration yourself, you own the entire API lifecycle. OpenPipe's API introduces several highly specific integration challenges that break standard LLM assumptions.

### The Endpoint Evolution and Deprecation Trap
OpenPipe is a fast-moving platform. Its API surface area has evolved rapidly as the platform has grown from a simple logging tool to a comprehensive fine-tuning pipeline. If you rely on an LLM's inherent training data to write API calls to OpenPipe, it will likely hallucinate requests to deprecated endpoints. 

For example, OpenPipe no longer supports prompt caching via the `/check-cache` endpoint. Additionally, older routing patterns like `/unstable/dataset/list` or `/finetune/create` have been aggressively deprecated in favor of standard RESTful resource paths like `/datasets` and `/models`. If your agent attempts to execute an outdated API call, it will hit a 404 Not Found error, causing the agent loop to crash unless you have written extensive error-handling and prompt-correction logic. You need an integration layer that dynamically maps the latest vendor schema into standardized Proxy APIs, removing the burden of maintaining endpoint mappings from your prompt context.

### Batch Limitations and Asynchronous State
When curating datasets for fine-tuning, you rarely add a single entry at a time. The OpenPipe API enforces a strict limit on the `/datasets/{dataset}/entries` endpoint - you can only submit a maximum of 100 entries per request. LLMs are notoriously bad at arbitrary chunking. If an agent tries to bulk-upload 500 generated interactions in a single network request, the OpenPipe API will reject the payload with a 400 Bad Request error. 

Furthermore, initiating a fine-tuning job via the `create_a_open_pipe_model` method is not a synchronous operation. The API returns a model ID and a pending status. The agent must understand that it needs to pause its operational loop, retain the model ID in its working memory, and periodically poll the `get_single_open_pipe_model_by_id` endpoint until the status changes from training to ready. Teaching an agent to manage asynchronous wait states requires specific system prompts and tool definitions that define exact expected inputs and outputs.

### The Reality of Rate Limits and Backoff Protocols
When your agent begins autonomously evaluating hundreds of completions using OpenPipe's criteria judging endpoints, it will inevitably hit HTTP 429 Too Many Requests errors. A critical architectural detail to understand: Truto does not retry, throttle, or apply backoff on rate limit errors. When the upstream OpenPipe API returns an HTTP 429, Truto passes that exact error directly back to the caller.

However, Truto normalizes the upstream rate limit information into standardized headers per the IETF specification. Regardless of how OpenPipe formats its rate limit headers natively, your agent framework will receive predictable `ratelimit-limit`, `ratelimit-remaining`, and `ratelimit-reset` headers. The caller - your agent loop or framework HTTP client - is entirely responsible for reading the reset timestamp, executing a sleep function, and retrying the request. This prevents the integration proxy from silently holding network connections open and gives your agent deterministic control over its execution speed.

## Fetching OpenPipe AI Agent Tools Programmatically

Instead of hardcoding OpenPipe endpoints into your codebase, Truto abstracts the API into a comprehensive JSON object that represents how the underlying product behaves. 

Integrations utilize a concept of `Resources` mapping to endpoints, enabling Truto to map any API into a REST-based CRUD API. Every Resource has `Methods` defined on them - standard operations like List, Get, Create, as well as custom operations. These Proxy APIs handle pagination, authentication, and query parameter processing, returning data in a predefined format.

Truto provides a set of tools for your LLM frameworks by offering a description and schema for all Methods defined on the Resources. By calling the `GET /integrated-account/<id>/tools` endpoint, you retrieve all these Proxy APIs pre-formatted for LLM consumption. 

Here is how you fetch and bind these tools using the Truto LangChain.js SDK:

```typescript
import { ChatOpenAI } from "@langchain/openai";
import { TrutoToolManager } from "truto-langchainjs-toolset";

// Initialize the LLM
const llm = new ChatOpenAI({
  modelName: "gpt-4o",
  temperature: 0,
});

// Initialize the Truto Tool Manager with your integrated OpenPipe account ID
const toolManager = new TrutoToolManager({
  trutoApiKey: process.env.TRUTO_API_KEY,
  integratedAccountId: "your-openpipe-integrated-account-id",
});

async function buildOpenPipeAgent() {
  // Fetch all available OpenPipe tools dynamically
  // You can filter by methods, e.g., methods: ['write', 'custom']
  const tools = await toolManager.getTools();

  // Bind the tools to the LLM natively
  const agentWithTools = llm.bindTools(tools);

  return agentWithTools;
}
```

This approach means your agent always has the correct, latest schema for OpenPipe without you writing a single fetch request.

## High-Leverage OpenPipe Tools for AI Agents

Below are the highest-leverage operations you can expose to an agent when managing OpenPipe workflows. Give your agent these capabilities to automate the entire lifecycle of model fine-tuning. 

### Create Chat Completion
**Tool:** `create_a_open_pipe_chat_completion`
This tool allows the agent to generate a chat completion directly through OpenPipe, which simultaneously logs the interaction for future dataset curation. It requires the `messages` array and the `model` identifier. 

> "Generate a response to the user query regarding password resets using the 'gpt-4o-mini' model via OpenPipe, ensuring the completion is tracked in our project logs."

### Record Request Log (Report)
**Tool:** `create_a_open_pipe_report`
Use this tool to manually record a request log from an OpenAI model call into OpenPipe. This is critical when your agent routes requests to standard OpenAI endpoints but needs to shadow-log specific high-value interactions into OpenPipe for future fine-tuning.

> "Take the prompt and completion from my last interaction with the user and log it to OpenPipe using the report tool. Tag it with 'intent:refund' and 'status:success'."

### Judge Completion Criteria
**Tool:** `create_a_open_pipe_criteria_judge`
This tool enables the agent to act as an evaluator, judging a specific completion against a pre-defined OpenPipe criterion. It returns a score, an explanation, and usage metrics, which is highly useful for automated dataset quality assurance.

> "Evaluate this generated customer support response against criterion ID 'crit_123abc' to determine if it meets our strict tone and formatting guidelines. Provide the resulting score and explanation."

### Create Dataset
**Tool:** `create_a_open_pipe_dataset`
This tool creates a new, empty dataset in OpenPipe. It requires a `name` and returns the dataset metadata including the ID, which the agent must store in context to append entries in subsequent steps.

> "Create a new dataset in OpenPipe named 'Support-Routing-Q3-Golden' and confirm the dataset ID so we can begin populating it with verified logs."

### Create Dataset Entries
**Tool:** `create_a_open_pipe_dataset_entry`
This tool creates new dataset entries in an existing OpenPipe dataset. It requires the `dataset_id` and an `entries` array. Note that the agent must limit the array to a maximum of 100 entries per request to avoid validation errors.

> "Take these 45 successful support logs we just extracted, format them as dataset entries, and add them to dataset ID 'ds_789xyz'. If any errors are returned in the creation summary, list the failed entry indexes."

### Train New Model
**Tool:** `create_a_open_pipe_model`
This tool initiates a fine-tuning job in OpenPipe. It requires the `datasetId`, a `slug` for the model name, and a `trainingConfig` object. The agent will receive the newly created model metadata, which will indicate a pending status.

> "Start a fine-tuning job on dataset ID 'ds_789xyz' using a base model of 'llama-3'. Set the model slug to 'support-router-v2' and return the new model ID."

To view the complete schema details, request formats, and the full list of supported operations - including dataset deletion, metadata updates, and model listings - view the [OpenPipe integration page](https://truto.one/integrations/detail/openpipe).

## Workflows in Action

Exposing individual tools to an LLM is only the first step. The true value of AI agents emerges when they chain multiple OpenPipe API calls together to execute complex operational workflows autonomously. 

### Scenario 1: Automated Completion Evaluation and Logging
**Persona:** Machine Learning Engineer

> "Fetch the last 10 raw customer interactions from our internal database, run each through our OpenPipe criteria judge for hallucination detection, and if the score is perfect, log the prompt and completion to OpenPipe as a verified request log."

**Agent Execution Steps:**
1. The agent fetches internal data (via an external database tool or custom logic).
2. The agent loops through the interactions, calling `create_a_open_pipe_criteria_judge` for each prompt/completion pair.
3. The agent parses the returned `score` and `explanation` from the judge.
4. For every interaction that receives a perfect score, the agent calls `create_a_open_pipe_report` to record the log securely into OpenPipe with a verified tag.

**Output:** The engineer receives a clean, automated evaluation loop that continually populates OpenPipe with high-quality, verified training data, eliminating the need for manual prompt review.

### Scenario 2: Autonomous Fine-Tuning Pipeline
**Persona:** DevOps / MLOps Administrator

> "Create a new dataset in OpenPipe called 'Nightly-Tone-Correction'. Take this JSON array of 85 corrected prompt-response pairs, format them, and add them to the new dataset. Once added, initiate a fine-tuning job using our standard training config."

**Agent Execution Steps:**
1. The agent calls `create_a_open_pipe_dataset` with the name "Nightly-Tone-Correction".
2. The agent extracts the `id` from the response.
3. The agent formats the provided JSON array into the required entry schema and calls `create_a_open_pipe_dataset_entry` using the new dataset ID, ensuring the batch size is under the 100-item limit.
4. Upon verifying the `entries_created` count matches the input, the agent calls `create_a_open_pipe_model` passing the dataset ID and configuration to start the training job.

**Output:** The administrator receives confirmation of the new dataset creation, verification that all 85 entries were successfully uploaded, and the ID of the pending fine-tuned model ready for tracking.

## Building Multi-Step Workflows

To orchestrate these multi-step processes reliably, your agent framework must handle the realities of API interaction, including pagination limits, strict payload shapes, and rate limits. Because Truto acts as a transparent proxy, standard HTTP 429 Too Many Requests errors are passed directly to your framework alongside normalized `ratelimit-reset` headers.

Here is an architectural view of how an agent handles an OpenPipe rate limit during a batch dataset ingestion process:

```mermaid
sequenceDiagram
    participant Agent as Agent Framework
    participant Truto as Truto Proxy
    participant OpenPipe as OpenPipe API
    
    Agent->>Truto: Call create_dataset_entry (Batch 1)
    Truto->>OpenPipe: POST /datasets/{id}/entries
    OpenPipe-->>Truto: HTTP 200 OK
    Truto-->>Agent: Success Response
    
    Agent->>Truto: Call create_dataset_entry (Batch 2)
    Truto->>OpenPipe: POST /datasets/{id}/entries
    OpenPipe-->>Truto: HTTP 429 Too Many Requests
    Truto-->>Agent: HTTP 429 (ratelimit-reset header)
    
    Note over Agent: Read reset header<br>Execute sleep function
    
    Agent->>Truto: Retry Call create_dataset_entry (Batch 2)
    Truto->>OpenPipe: POST /datasets/{id}/entries
    OpenPipe-->>Truto: HTTP 200 OK
    Truto-->>Agent: Success Response
```

To implement this within an AI Agent loop, you must ensure your tool execution logic wraps calls in a retry block that respects these normalized headers. While Truto normalizes the schema mapping - ensuring that a `datasetId` mapping is always structurally correct - the responsibility for workflow resilience remains with the agent.

When using LangChain, [LangGraph](https://truto.one/architecting-ai-agents-langgraph-langchain-and-the-saas-integration-bottleneck/), or Vercel AI SDK, you can intercept tool errors and feed them back to the LLM as observation steps, allowing the model to self-correct payload structures if it hallucinates an invalid parameter.

```typescript
// Example of feeding tool execution back to the agent in LangGraph
import { ToolNode } from "@langchain/langgraph/prebuilt";

// We pass our dynamically fetched Truto tools into the ToolNode
const toolNode = new ToolNode(tools);

// Inside your graph definition, if a tool fails (e.g., HTTP 400 Bad Request due to >100 entries),
// the error string is returned as the tool output. 
// The LLM sees the error: "Max 100 entries per request" and can autonomously chunk the array.
```

By leveraging Proxy APIs that handle the boilerplate of authentication and schema parsing, your agent can focus entirely on the business logic of dataset curation and model evaluation.

> Stop wasting engineering cycles maintaining custom connectors and brittle polling logic. Give your AI agents reliable, normalized access to OpenPipe and 200+ other enterprise SaaS applications with Truto.
>
> [Talk to us](https://cal.com/truto/partner-with-truto)

Connecting OpenPipe to an AI agent doesn't require maintaining a massive repository of custom endpoint wrappers. By utilizing Truto's `/tools` endpoint, you dynamically load heavily typed, LLM-ready functions directly into your agent framework. This architecture decouples your agent's reasoning capabilities from the underlying complexities of API schema changes, endpoint deprecations, and authentication headers, allowing you to build autonomous machine learning operations pipelines that simply work.