Connect OpenPipe to AI Agents: Automate Dataset Curation and Fine-Tuning

You want to connect OpenPipe to an AI agent so your internal systems can independently evaluate completions, curate training datasets, execute fine-tuning jobs, and manage custom models based on historical application logs. Here is exactly how to do it using Truto's /tools endpoint and SDK, bypassing the need to manually code API wrappers or maintain complex polling logic for asynchronous training jobs.

Giving a Large Language Model (LLM) read and write access to your OpenPipe instance is an engineering headache. You either spend weeks building, hosting, and maintaining a custom connector that understands the nuances of dataset batching and legacy endpoint deprecations, or you use a managed infrastructure layer that handles the boilerplate for you. If your team uses ChatGPT, check out our guide on connecting OpenPipe to ChatGPT, or if you are building on Anthropic's models, read our guide on connecting OpenPipe to Claude. For developers building custom autonomous workflows, you need a programmatic way to fetch these tools and bind them to your agent framework.

This guide breaks down exactly how to fetch AI-ready tools for OpenPipe, bind them natively to an LLM using LangChain (or any framework like LangGraph, CrewAI, or Vercel AI SDK), and execute complex dataset curation and fine-tuning workflows. For a deeper look at the architecture behind this approach, refer to our research on architecting AI agents and the SaaS integration bottleneck.

The Engineering Reality of Custom OpenPipe Connectors

Building AI agents is easy. Connecting them to external SaaS APIs is hard. Giving an LLM access to external data sounds simple in a prototype. You write a Node.js function that makes a fetch request and wrap it in an @tool decorator. In production, this approach collapses entirely, especially with a specialized machine learning operations platform like OpenPipe.

If you decide to build a custom OpenPipe integration yourself, you own the entire API lifecycle. OpenPipe's API introduces several highly specific integration challenges that break standard LLM assumptions.

The Endpoint Evolution and Deprecation Trap

OpenPipe is a fast-moving platform. Its API surface area has evolved rapidly as the platform has grown from a simple logging tool to a comprehensive fine-tuning pipeline. If you rely on an LLM's inherent training data to write API calls to OpenPipe, it will likely hallucinate requests to deprecated endpoints.

For example, OpenPipe no longer supports prompt caching via the /check-cache endpoint. Additionally, older routing patterns like /unstable/dataset/list or /finetune/create have been aggressively deprecated in favor of standard RESTful resource paths like /datasets and /models. If your agent attempts to execute an outdated API call, it will hit a 404 Not Found error, causing the agent loop to crash unless you have written extensive error-handling and prompt-correction logic. You need an integration layer that dynamically maps the latest vendor schema into standardized Proxy APIs, removing the burden of maintaining endpoint mappings from your prompt context.

Batch Limitations and Asynchronous State

When curating datasets for fine-tuning, you rarely add a single entry at a time. The OpenPipe API enforces a strict limit on the /datasets/{dataset}/entries endpoint - you can only submit a maximum of 100 entries per request. LLMs are notoriously bad at arbitrary chunking. If an agent tries to bulk-upload 500 generated interactions in a single network request, the OpenPipe API will reject the payload with a 400 Bad Request error.

Furthermore, initiating a fine-tuning job via the create_a_open_pipe_model method is not a synchronous operation. The API returns a model ID and a pending status. The agent must understand that it needs to pause its operational loop, retain the model ID in its working memory, and periodically poll the get_single_open_pipe_model_by_id endpoint until the status changes from training to ready. Teaching an agent to manage asynchronous wait states requires specific system prompts and tool definitions that define exact expected inputs and outputs.

The Reality of Rate Limits and Backoff Protocols

When your agent begins autonomously evaluating hundreds of completions using OpenPipe's criteria judging endpoints, it will inevitably hit HTTP 429 Too Many Requests errors. A critical architectural detail to understand: Truto does not retry, throttle, or apply backoff on rate limit errors. When the upstream OpenPipe API returns an HTTP 429, Truto passes that exact error directly back to the caller.

However, Truto normalizes the upstream rate limit information into standardized headers per the IETF specification. Regardless of how OpenPipe formats its rate limit headers natively, your agent framework will receive predictable ratelimit-limit, ratelimit-remaining, and ratelimit-reset headers. The caller - your agent loop or framework HTTP client - is entirely responsible for reading the reset timestamp, executing a sleep function, and retrying the request. This prevents the integration proxy from silently holding network connections open and gives your agent deterministic control over its execution speed.

Fetching OpenPipe AI Agent Tools Programmatically

Instead of hardcoding OpenPipe endpoints into your codebase, Truto abstracts the API into a comprehensive JSON object that represents how the underlying product behaves.

Integrations utilize a concept of Resources mapping to endpoints, enabling Truto to map any API into a REST-based CRUD API. Every Resource has Methods defined on them - standard operations like List, Get, Create, as well as custom operations. These Proxy APIs handle pagination, authentication, and query parameter processing, returning data in a predefined format.

Truto provides a set of tools for your LLM frameworks by offering a description and schema for all Methods defined on the Resources. By calling the GET /integrated-account/<id>/tools endpoint, you retrieve all these Proxy APIs pre-formatted for LLM consumption.

Here is how you fetch and bind these tools using the Truto LangChain.js SDK:

import { ChatOpenAI } from "@langchain/openai";
import { TrutoToolManager } from "truto-langchainjs-toolset";
 
// Initialize the LLM
const llm = new ChatOpenAI({
  modelName: "gpt-4o",
  temperature: 0,
});
 
// Initialize the Truto Tool Manager with your integrated OpenPipe account ID
const toolManager = new TrutoToolManager({
  trutoApiKey: process.env.TRUTO_API_KEY,
  integratedAccountId: "your-openpipe-integrated-account-id",
});
 
async function buildOpenPipeAgent() {
  // Fetch all available OpenPipe tools dynamically
  // You can filter by methods, e.g., methods: ['write', 'custom']
  const tools = await toolManager.getTools();
 
  // Bind the tools to the LLM natively
  const agentWithTools = llm.bindTools(tools);
 
  return agentWithTools;
}

This approach means your agent always has the correct, latest schema for OpenPipe without you writing a single fetch request.

High-Leverage OpenPipe Tools for AI Agents

Below are the highest-leverage operations you can expose to an agent when managing OpenPipe workflows. Give your agent these capabilities to automate the entire lifecycle of model fine-tuning.

Create Chat Completion

Tool: create_a_open_pipe_chat_completion This tool allows the agent to generate a chat completion directly through OpenPipe, which simultaneously logs the interaction for future dataset curation. It requires the messages array and the model identifier.

"Generate a response to the user query regarding password resets using the 'gpt-4o-mini' model via OpenPipe, ensuring the completion is tracked in our project logs."

Record Request Log (Report)

Tool: create_a_open_pipe_report Use this tool to manually record a request log from an OpenAI model call into OpenPipe. This is critical when your agent routes requests to standard OpenAI endpoints but needs to shadow-log specific high-value interactions into OpenPipe for future fine-tuning.

"Take the prompt and completion from my last interaction with the user and log it to OpenPipe using the report tool. Tag it with 'intent:refund' and 'status:success'."

Judge Completion Criteria

Tool: create_a_open_pipe_criteria_judge This tool enables the agent to act as an evaluator, judging a specific completion against a pre-defined OpenPipe criterion. It returns a score, an explanation, and usage metrics, which is highly useful for automated dataset quality assurance.

"Evaluate this generated customer support response against criterion ID 'crit_123abc' to determine if it meets our strict tone and formatting guidelines. Provide the resulting score and explanation."

Create Dataset

Tool: create_a_open_pipe_dataset This tool creates a new, empty dataset in OpenPipe. It requires a name and returns the dataset metadata including the ID, which the agent must store in context to append entries in subsequent steps.

"Create a new dataset in OpenPipe named 'Support-Routing-Q3-Golden' and confirm the dataset ID so we can begin populating it with verified logs."

Create Dataset Entries

Tool: create_a_open_pipe_dataset_entry This tool creates new dataset entries in an existing OpenPipe dataset. It requires the dataset_id and an entries array. Note that the agent must limit the array to a maximum of 100 entries per request to avoid validation errors.

"Take these 45 successful support logs we just extracted, format them as dataset entries, and add them to dataset ID 'ds_789xyz'. If any errors are returned in the creation summary, list the failed entry indexes."

Train New Model

Tool: create_a_open_pipe_model This tool initiates a fine-tuning job in OpenPipe. It requires the datasetId, a slug for the model name, and a trainingConfig object. The agent will receive the newly created model metadata, which will indicate a pending status.

"Start a fine-tuning job on dataset ID 'ds_789xyz' using a base model of 'llama-3'. Set the model slug to 'support-router-v2' and return the new model ID."

To view the complete schema details, request formats, and the full list of supported operations - including dataset deletion, metadata updates, and model listings - view the OpenPipe integration page.

Workflows in Action

Exposing individual tools to an LLM is only the first step. The true value of AI agents emerges when they chain multiple OpenPipe API calls together to execute complex operational workflows autonomously.

Scenario 1: Automated Completion Evaluation and Logging

Persona: Machine Learning Engineer

"Fetch the last 10 raw customer interactions from our internal database, run each through our OpenPipe criteria judge for hallucination detection, and if the score is perfect, log the prompt and completion to OpenPipe as a verified request log."

Agent Execution Steps:

1. The agent fetches internal data (via an external database tool or custom logic).
2. The agent loops through the interactions, calling create_a_open_pipe_criteria_judge for each prompt/completion pair.
3. The agent parses the returned score and explanation from the judge.
4. For every interaction that receives a perfect score, the agent calls create_a_open_pipe_report to record the log securely into OpenPipe with a verified tag.

Output: The engineer receives a clean, automated evaluation loop that continually populates OpenPipe with high-quality, verified training data, eliminating the need for manual prompt review.

Scenario 2: Autonomous Fine-Tuning Pipeline

Persona: DevOps / MLOps Administrator

"Create a new dataset in OpenPipe called 'Nightly-Tone-Correction'. Take this JSON array of 85 corrected prompt-response pairs, format them, and add them to the new dataset. Once added, initiate a fine-tuning job using our standard training config."

Agent Execution Steps:

1. The agent calls create_a_open_pipe_dataset with the name "Nightly-Tone-Correction".
2. The agent extracts the id from the response.
3. The agent formats the provided JSON array into the required entry schema and calls create_a_open_pipe_dataset_entry using the new dataset ID, ensuring the batch size is under the 100-item limit.
4. Upon verifying the entries_created count matches the input, the agent calls create_a_open_pipe_model passing the dataset ID and configuration to start the training job.

Output: The administrator receives confirmation of the new dataset creation, verification that all 85 entries were successfully uploaded, and the ID of the pending fine-tuned model ready for tracking.

Building Multi-Step Workflows

To orchestrate these multi-step processes reliably, your agent framework must handle the realities of API interaction, including pagination limits, strict payload shapes, and rate limits. Because Truto acts as a transparent proxy, standard HTTP 429 Too Many Requests errors are passed directly to your framework alongside normalized ratelimit-reset headers.

Here is an architectural view of how an agent handles an OpenPipe rate limit during a batch dataset ingestion process:

sequenceDiagram
    participant Agent as Agent Framework
    participant Truto as Truto Proxy
    participant OpenPipe as OpenPipe API
    
    Agent->>Truto: Call create_dataset_entry (Batch 1)
    Truto->>OpenPipe: POST /datasets/{id}/entries
    OpenPipe-->>Truto: HTTP 200 OK
    Truto-->>Agent: Success Response
    
    Agent->>Truto: Call create_dataset_entry (Batch 2)
    Truto->>OpenPipe: POST /datasets/{id}/entries
    OpenPipe-->>Truto: HTTP 429 Too Many Requests
    Truto-->>Agent: HTTP 429 (ratelimit-reset header)
    
    Note over Agent: Read reset header<br>Execute sleep function
    
    Agent->>Truto: Retry Call create_dataset_entry (Batch 2)
    Truto->>OpenPipe: POST /datasets/{id}/entries
    OpenPipe-->>Truto: HTTP 200 OK
    Truto-->>Agent: Success Response

To implement this within an AI Agent loop, you must ensure your tool execution logic wraps calls in a retry block that respects these normalized headers. While Truto normalizes the schema mapping - ensuring that a datasetId mapping is always structurally correct - the responsibility for workflow resilience remains with the agent.

When using LangChain, LangGraph, or Vercel AI SDK, you can intercept tool errors and feed them back to the LLM as observation steps, allowing the model to self-correct payload structures if it hallucinates an invalid parameter.

// Example of feeding tool execution back to the agent in LangGraph
import { ToolNode } from "@langchain/langgraph/prebuilt";
 
// We pass our dynamically fetched Truto tools into the ToolNode
const toolNode = new ToolNode(tools);
 
// Inside your graph definition, if a tool fails (e.g., HTTP 400 Bad Request due to >100 entries),
// the error string is returned as the tool output. 
// The LLM sees the error: "Max 100 entries per request" and can autonomously chunk the array.

By leveraging Proxy APIs that handle the boilerplate of authentication and schema parsing, your agent can focus entirely on the business logic of dataset curation and model evaluation.

Connecting OpenPipe to an AI agent doesn't require maintaining a massive repository of custom endpoint wrappers. By utilizing Truto's /tools endpoint, you dynamically load heavily typed, LLM-ready functions directly into your agent framework. This architecture decouples your agent's reasoning capabilities from the underlying complexities of API schema changes, endpoint deprecations, and authentication headers, allowing you to build autonomous machine learning operations pipelines that simply work.