Connect Portkey to AI Agents: Automate Fine-Tuning, OCR & Threads

You want to connect Portkey to an AI agent so your system can independently orchestrate AI gateways, trigger fine-tuning jobs, extract documents via OCR, and manage complex conversation threads based on real-time observability data. Here is exactly how to do it using Truto's /tools endpoint and SDK, bypassing the need to hand-code complex API wrappers for LLM operations.

Giving a Large Language Model (LLM) read and write access to your Portkey workspace introduces a massive engineering bottleneck. You either spend weeks building and maintaining a custom connector that understands Portkey's intricate configuration payloads and routing rules, or you use a managed infrastructure layer that handles the boilerplate for you. If your team uses ChatGPT, check out our guide on connecting Portkey to ChatGPT, or if you are building on Anthropic's models, read our guide on connecting Portkey to Claude. For developers building custom autonomous workflows, you need a programmatic way to fetch these tools and bind them to your agent framework.

This guide breaks down exactly how to fetch AI-ready tools for Portkey, bind them natively to an LLM using LangChain (or any framework like LangGraph, CrewAI, or Vercel AI SDK), and execute complex AI operations workflows. For a deeper look at the architecture behind this approach, refer to our research on architecting AI agents and the SaaS integration bottleneck.

The Engineering Reality of Custom Portkey Connectors

Building an AI agent is relatively straightforward until you need it to interact reliably with an external system. Giving an LLM access to external APIs sounds simple in a Jupyter Notebook prototype. You write a Node.js function that makes a fetch request and wrap it in a tool decorator. In production, this approach collapses entirely, especially with an ecosystem as complex as Portkey.

If you decide to build a direct Portkey integration yourself, you own the entire API lifecycle. Portkey is essentially a control plane and gateway for 100+ different AI models. Its API introduces highly specific integration challenges that break standard LLM assumptions.

The Hyperparameter Payload Trap

When interacting with prompt execution endpoints in Portkey (like rendering or completing a saved prompt), the API expects variables and hyperparameters to be structured in a very specific, non-intuitive way. Standard LLMs hallucinate JSON payloads by assuming properties like temperature or max_completion_tokens belong nested inside a config or hyperparameters object. Portkey explicitly requires these at the root level alongside the prompt variables. Furthermore, Portkey strictly deprecates older OpenAI conventions - passing max_tokens instead of max_completion_tokens will cause failures. If you hand-code this tool, you must write extensive, brittle prompt engineering just to teach the LLM the correct JSON structure for this specific vendor.

Thread State Machine Complexity

Portkey provides thread and run management (parity with the OpenAI Assistants API). When an agent needs to create and execute a thread run, it does not receive a synchronous, final output. Instead, the API returns a run object with a status like in_progress or requires_action. Your system must handle a polling loop. If an LLM is directly calling these endpoints, it often hallucinates the state progression, attempting to fetch a thread response before the status has transitioned to completed, or failing to format the required tool_outputs payload correctly when action is required.

Guardrail and Config ID Resolution

Portkey configurations, virtual keys, and guardrails use mixed identifier conventions. A guardrail might be referenced by a standard UUID or by a specific slug prefixed with guard_. When dynamically routing requests or updating workspace configurations, the LLM must perfectly recall which ID format to use for which resource type. Every minor hallucination results in a 400 Bad Request or a routing failure deep inside the AI Gateway.

Why a Unified Tool Layer Matters for Agent Safety

Before writing a single line of integration code, decide what architectural layer your agent will interface with. This choice dictates the safety and reliability of your production system.

Direct API tools push provider quirks directly into the LLM's context window. The model has to remember that Portkey requires root-level hyperparameters, that threads require state polling, and that guardrail IDs use custom prefixes. Every one of those quirks drains context tokens and increases the probability of a hallucinated API call.

A unified tool layer collapses these complexities behind standardized schemas. Your agent interfaces with clear, descriptive functions. This architecture provides critical engineering advantages:

1. Smaller attack surface for hallucination. The LLM only ever chooses from stable function names with deterministic, pre-validated JSON schemas. Invalid arguments are rejected at the schema level before they ever hit the Portkey API.
2. Schema isolation. When Portkey deprecates a field (e.g., swapping max_tokens for max_completion_tokens), the underlying proxy handles the mapping. Your agent's context and tool definitions remain stable.

A Note on Rate Limits and Reliability

When building autonomous agents, developers often assume the integration layer will magically absorb API limits. This is a dangerous architectural anti-pattern. Truto explicitly does not retry, throttle, or apply backoff on rate limit errors. When the upstream Portkey API returns an HTTP 429 Too Many Requests, Truto passes that error directly to the caller.

What Truto does is normalize the upstream rate limit information into standardized headers (ratelimit-limit, ratelimit-remaining, ratelimit-reset) per the IETF specification. It is strictly the responsibility of your agent loop (or the framework like LangGraph) to read the ratelimit-reset header, pause execution, and retry the tool call. This guarantees that your core application logic remains in control of scheduling and execution costs.

flowchart TD
    A["AI Agent Loop<br>(LangGraph/CrewAI)"] -->|"Executes Tool (JSON)"| B["Truto Unified Proxy"]
    B -->|"Validates Schema"| C{"Schema Valid?"}
    C -->|"Yes"| D["Portkey API"]
    C -->|"No"| E["Return 400 to Agent"]
    D -->|"HTTP 429 Rate Limit"| B
    B -->|"Passes 429 + Standard Headers"| A
    A -->|"Wait & Retry"| A

Portkey Hero Tools for AI Agents

Truto provides a comprehensive set of tools mapped to Portkey's API resources. Below are the highest-leverage hero tools for automating MLOps, AI gateways, and RAG pipelines.

1. Execute Prompt Completions (create_a_portkey_prompt_completion)

Executes a saved prompt template in Portkey to generate completions with variable substitution. The agent can trigger complex, pre-tested prompts stored in the workspace, drastically reducing the size of the payload it needs to construct on the fly. Hyperparameters must be dynamically injected at the root level.

"Run the 'extract-entities' prompt (ID: 8f7e6...) using the variable 'document_text' set to the contents of the latest support email. Set max_completion_tokens to 500 and temperature to 0.2."

2. Automate Fine-Tuning Jobs (create_a_portkey_fine_tuning_job)

Initiates a fine-tuning job in Portkey for a provider model. Agents can automatically trigger model training once a specific dataset threshold is reached or when performance metrics degrade below a configured SLA.

"Take the processed training file ID from the previous step and initiate a new fine-tuning job on the standard gpt-4o-mini model. Return the fine-tuning job ID so we can monitor its status."

3. Extract Document Structure (create_a_portkey_ocr)

Extracts text and structured content from complex documents (PDFs and images) using Portkey's OCR models. Crucial for agents building dynamic RAG pipelines from raw user uploads or messy internal knowledge bases.

"Send the invoice PDF located at this URL through the Portkey OCR endpoint using the Mistral AI vision model. Return the structured pages and token usage info."

4. Create and Execute Thread Runs (create_a_portkey_threads_run)

Creates a thread and immediately executes a run in a single request. This is the optimal tool for an agent to spin off asynchronous sub-tasks or manage isolated conversational contexts using a specific pre-configured Assistant ID.

"Initialize a new thread run using Assistant ID 'ast_9283...' and pass the user's initial message regarding their refund status. Keep the polling loop active until the status transitions out of in_progress."

5. Update Gateway Configurations (update_a_portkey_config_by_id)

Modifies a Portkey configuration dynamically. An observability agent can use this tool to rewrite fallback strategies, adjust caching TTLs, or change routing targets based on real-time provider latency or cost spikes.

"Update the routing configuration ID 'cfg_123...' to change the primary target model to Claude 3.5 Sonnet, and set the fallback target to GPT-4o. Ensure cache is set to semantic mode."

6. Manage Guardrails (create_a_portkey_guardrail)

Creates a new guardrail in Portkey with specified validation checks and actions. Agents can dynamically deploy safety policies based on the type of data they are about to process, ensuring compliance and preventing jailbreaks.

"Create a new guardrail named 'Strict PII Filter'. Add checks for email addresses, phone numbers, and SSNs. Set the action to block the request entirely if any check fails."

To view the complete inventory of available tools and their exact JSON schemas, visit the Portkey integration page.

Workflows in Action

How do these tools compose into actual autonomous operations? Here are two concrete, persona-specific examples of multi-step agent workflows.

Scenario 1: Automated Model Evaluation & Guardrail Deployment

Persona: MLOps Engineer / AI Architect Trigger: A scheduled agent script runs weekly to evaluate the safety metrics of the primary customer-facing AI Assistant.

"Analyze the error logs from the past 7 days. If the rate of flagged inappropriate requests exceeds 2%, create a new Portkey Guardrail that checks for toxic language and jailbreak attempts. Then, update the primary production Configuration to enforce this new guardrail immediately."

Tool Execution Flow:

1. list_all_portkey_graphs_errors: The agent fetches the aggregated error data and analyzes the status codes and metadata for safety violations.
2. create_a_portkey_guardrail: Determining a spike in toxicity, the agent constructs a payload with specific checks (e.g., toxic language, prompt injection) and actions (block), receiving a new guard_ prefixed ID.
3. update_a_portkey_config_by_id: The agent patches the main production routing config, injecting the new guardrail ID into the execution path to protect all subsequent LLM requests.

Result: The system independently detects a safety degradation and deploys a strict, provider-agnostic filter at the gateway level without human intervention.

Scenario 2: Dynamic RAG & OCR Pipeline

Persona: Data Engineer / Support Ops Trigger: A batch of legacy PDF contracts is dropped into a monitored cloud bucket.

"Process the uploaded PDF contracts. Run them through Portkey's OCR models to extract the raw text. Then, create a new Portkey File with the output, and attach it to the 'Enterprise Contracts' Vector Store so our sales assistant can query it."

Tool Execution Flow:

1. create_a_portkey_ocr: The agent passes the raw documents to the OCR endpoint, receiving high-quality, structured text extraction.
2. create_a_portkey_file: The agent packages the extracted text into a clean JSONL/TXT format and uploads it to Portkey with the purpose set to assistants.
3. create_a_portkey_vector_store_file: The agent takes the returned file ID and attaches it to the target vector store ID, triggering Portkey's chunking and embedding processes.

Result: Unstructured, image-heavy PDFs are automatically parsed, embedded, and made instantly searchable by downstream AI assistants.

Building Multi-Step Workflows

To execute these complex loops, you need to bind Truto's dynamically generated proxy tools to your agent framework. The following architecture works across LangChain, LangGraph, CrewAI, and the Vercel AI SDK.

We will use the @langchain/core and truto-langchainjs-toolset packages to demonstrate fetching Portkey tools and handling the critical rate limit retry logic.

import { ChatOpenAI } from "@langchain/openai";
import { TrutoToolManager } from "truto-langchainjs-toolset";
import { HumanMessage } from "@langchain/core/messages";
 
async function runPortkeyAgent(prompt: string) {
  // 1. Initialize the LLM
  const llm = new ChatOpenAI({
    modelName: "gpt-4o",
    temperature: 0,
  });
 
  // 2. Initialize Truto Tool Manager for the Portkey Integrated Account
  // This fetches the schema-validated tools dynamically
  const truto = new TrutoToolManager({
    apiKey: process.env.TRUTO_API_KEY,
    integratedAccountId: process.env.PORTKEY_ACCOUNT_ID,
  });
 
  // 3. Fetch specific Hero Tools based on the workflow
  const tools = await truto.getTools({
    methods: ["create_a_portkey_ocr", "create_a_portkey_guardrail", "update_a_portkey_config_by_id"]
  });
 
  // 4. Bind tools to the LLM
  const llmWithTools = llm.bindTools(tools);
 
  let messages = [new HumanMessage(prompt)];
  let isComplete = false;
 
  // 5. Agent Execution Loop
  while (!isComplete) {
    const response = await llmWithTools.invoke(messages);
    messages.push(response);
 
    if (!response.tool_calls || response.tool_calls.length === 0) {
      console.log("Agent finished:", response.content);
      isComplete = true;
      break;
    }
 
    // 6. Execute Tool Calls with Rate Limit Handling
    for (const toolCall of response.tool_calls) {
      const selectedTool = tools.find((t) => t.name === toolCall.name);
      if (!selectedTool) continue;
 
      try {
        console.log(`Executing ${toolCall.name}...`);
        const toolResult = await selectedTool.invoke(toolCall.args);
        
        messages.push({
          role: "tool",
          content: JSON.stringify(toolResult),
          tool_call_id: toolCall.id,
        });
 
      } catch (error: any) {
        // STRICT REQUIREMENT: Truto passes 429s back. The caller must handle it.
        if (error.response && error.response.status === 429) {
            const resetSeconds = error.response.headers['ratelimit-reset'] || 5;
            console.warn(`Rate limited by Portkey. Retrying in ${resetSeconds} seconds...`);
            
            await new Promise(resolve => setTimeout(resolve, resetSeconds * 1000));
            
            // Retry logic would go here in a robust system (e.g., pushing the tool call back onto a queue)
            messages.push({
                role: "tool",
                content: JSON.stringify({ error: "Rate limited. Will retry on next loop." }),
                tool_call_id: toolCall.id,
            });
        } else {
            messages.push({
                role: "tool",
                content: JSON.stringify({ error: error.message }),
                tool_call_id: toolCall.id,
            });
        }
      }
    }
  }
}
 
runPortkeyAgent("Check error logs and create a new guardrail for toxicity if errors are high, then update config cfg_abc123.");

Architectural Sequence

When the agent initiates a sequence, Truto acts as the enforcement layer between the LLM and Portkey.

sequenceDiagram
    participant LLM as "Agent (LangChain)"
    participant Truto as "Truto Tools Layer"
    participant Portkey as "Portkey API"

    LLM->>Truto: fetch tools for Portkey
    Truto-->>LLM: returns JSON schemas (Proxy APIs)
    LLM->>LLM: bindTools()
    
    Note over LLM,Portkey: Agent Loop Begins
    LLM->>Truto: execute create_a_portkey_guardrail(args)
    Truto->>Truto: Validate JSON schema
    Truto->>Portkey: POST /v1/guardrails
    
    alt API Rate Limit Hit
        Portkey-->>Truto: 429 Too Many Requests
        Truto-->>LLM: 429 Error + ratelimit-reset header
        LLM->>LLM: Exponential Backoff (Agent Logic)
    else Success
        Portkey-->>Truto: 200 OK (Guardrail ID)
        Truto-->>LLM: Standardized Success Payload
    end

Strategic Wrap-Up

Connecting AI agents to Portkey transforms your system from a passive observer of LLM traffic into an active controller of AI operations. By bypassing the fragile nature of direct API integration and utilizing a unified tool layer, your engineering team fundamentally changes how reliable the agent becomes.

You isolate the LLM from Portkey's payload quirks. You shrink the context window by providing deterministic, validated schemas. And you maintain total architectural control over critical infrastructure mechanisms like rate limiting and retries, utilizing standard IETF headers passed straight back to your agent framework.

Stop writing custom wrappers and prompt engineering your way around bad JSON. Expose Portkey as a set of robust tools and let the agent do the heavy lifting.