Connect LangChain to AI Agents: Track Performance and Feedback

You want to connect the LangChain and LangSmith ecosystem to an AI agent so your system can independently track trace performance, monitor evaluation metrics, curate golden datasets, and inject automated feedback into your observability pipeline. Here is exactly how to do it using Truto's /tools endpoint and SDK, bypassing the need to build and maintain custom REST API wrappers for LangSmith from scratch.

Giving a Large Language Model (LLM) read and write access to your LangSmith deployment is an engineering challenge. You either spend sprints studying the API documentation to write strict JSON schemas for your agent, or you use a managed infrastructure layer that provides these LLM-ready definitions out of the box. If your team uses ChatGPT, check out our guide on connecting LangChain to ChatGPT, or if you are building on Anthropic's models, read our guide on connecting LangChain to Claude. For developers building custom autonomous workflows - whether you are using LangGraph, CrewAI, or the Vercel AI SDK - you need a programmatic way to fetch these tools and bind them to your framework.

This guide breaks down exactly how to fetch AI-ready tools for the LangChain ecosystem, bind them natively to your LLM framework of choice, and execute complex observability and tracing workflows. For a deeper look at the architecture behind this approach, refer to our research on architecting AI agents and the SaaS integration bottleneck.

Why a Unified Tool Layer Matters for Observability

Before writing a line of code to parse trace data, decide what layer your agent will talk to. This choice determines the reliability of your observability workflows in production.

Direct API tools (mapping one raw LangSmith endpoint directly to an LLM tool) push provider quirks right into the model's context window. The agent is forced to understand exactly how LangSmith structures its filtering grammar, how it nests metadata payloads, and how it handles execution orders. Every API idiosyncrasy is a hallucination waiting to happen.

A unified tool layer abstracts these endpoints into deterministic, schema-bound functions. Your agent simply sees list_all_lang_chain_runs_queries or create_a_lang_chain_feedback. This provides clear engineering advantages:

1. Minimized hallucination surface area. The agent only selects from a tightly bounded list of stable function names. It does not attempt to construct raw HTTP POST bodies.
2. Deterministic input validation. Every tool provided by Truto has a strict JSON schema mapped from the underlying API. Invalid arguments are rejected locally before the network request is even initiated, failing fast rather than failing halfway through a complex data operation.

The Engineering Reality of the LangSmith API

Giving an LLM access to external trace data sounds simple during prototyping. In a production environment, this approach quickly breaks down. The LangSmith API introduces specific integration challenges that break standard REST assumptions. If you hardcode these interactions into your agent, you will spend your time writing defensive integration code instead of improving your application.

Trace Filter String Complexity

When querying runs and traces, LangSmith relies on complex filter string syntax. You cannot simply pass a flat query parameter like ?status=error. The API requires structured, encoded filter expressions such as eq(status, "error") or deep JSON path matching for metadata fields. If you expose the raw REST interface to an LLM, the model will frequently guess the filter syntax incorrectly. Truto's tools encapsulate these schemas, but you must still provide the LLM with clear instructions on how to format specific filter criteria for the list_all_lang_chain_runs_queries tool.

Highly Dynamic I/O Payloads

LangSmith is designed to log the inputs and outputs of any LLM chain. Because of this, the inputs and outputs fields inside a run payload are entirely dynamic JSON objects. They have no fixed schema - one trace might contain a flat string, while another contains an array of nested message objects. When passing these dynamic JSON blobs back into an agent loop for evaluation, the agent can become confused by the unpredictable structure. You must strictly constrain the agent's extraction logic when dealing with these open-ended fields.

Rate Limiting and Transparent Proxies

When your agent begins iterating over thousands of traces to run evaluations, you will inevitably hit LangSmith rate limits. It is critical to understand that Truto does not retry, throttle, or apply backoff on rate limit errors.

When the upstream LangSmith API returns an HTTP 429 Too Many Requests, Truto passes that error directly to the caller. However, Truto normalizes the upstream rate limit information into standardized headers (ratelimit-limit, ratelimit-remaining, ratelimit-reset) per the IETF specification. This design ensures your application retains full control over execution pausing. Your agent execution loop is solely responsible for reading the ratelimit-reset header, sleeping the thread, and retrying. Do not expect the integration layer to magically absorb high-volume API rejections.

High-Leverage Tools for LangChain

Truto provides a comprehensive set of tools for LangChain and LangSmith. Rather than exposing generic CRUD operations, you should arm your agent with the specific endpoints needed to orchestrate observability and RAG workflows. Here are the most critical tools to bind to your LLM.

List All LangSmith Sessions

Before an agent can analyze runs, it needs to know which project (session) to look at. This tool returns the available tracer sessions along with top-level statistics like latency percentiles and token costs.

"Find the session ID for our 'production-support-bot' project and give me the p99 latency for its recent runs."

Query LangSmith Runs

This is the core tool for retrieving specific traces. It accepts filter expressions and time boundaries, allowing the agent to hunt down specific execution paths, errors, or slow LLM calls.

"Query the runs in the 'customer-onboarding' session from the last 24 hours where the execution status was an error, and return the trace IDs."

Get Single Run by ID

Once a problematic trace is identified, the agent uses this tool to pull the deep execution tree. This returns the exact inputs, outputs, error stack traces, and token usage for a single specific operation.

"Retrieve the full details for run ID 8f9b2a1c... and tell me exactly what prompt was sent to the model right before the failure."

Create LangSmith Feedback

This tool enables autonomous LLM-as-a-judge workflows. After an agent analyzes a run's output against a specific rubric, it can use this tool to post a score (like relevance, toxicity, or helpfulness) directly back to the trace in LangSmith.

"Evaluate the output of run ID 3a2c9b... for tone. If it is aggressive, log a feedback score of 0 with the key 'tone_check' and include your reasoning in the comment."

List All Datasets

For dataset curation, the agent needs to locate the correct target dataset. This tool lists all available datasets in the workspace, providing the IDs needed for appending new examples.

"List all datasets and find the ID for the dataset named 'Golden-RAG-Retrievals'."

Create Dataset Example

When your agent discovers an edge case in production that the LLM handled perfectly, it can use this tool to extract the inputs and outputs from the trace and permanently save them as a fine-tuning or evaluation example.

"Take the inputs and outputs from the successful run ID 7d4e1f... and add them as a new example to the 'Golden-RAG-Retrievals' dataset."

For the complete inventory of available endpoints and their exact JSON schema definitions, visit the LangChain integration page.

Workflows in Action

Giving an agent access to these tools transforms static observability into an active, self-healing system. Here are two concrete ways engineering teams deploy this architecture.

Scenario 1: Autonomous LLM-as-a-Judge

Running manual evaluations on every production chat interaction is impossible. Teams use AI agents to sample production logs, evaluate them against a strict rubric, and write scores back to the observability platform.

"Review the last 10 traces in the 'Support-Copilot' session. For each trace, read the user input and the bot's final output. Evaluate if the bot successfully answered the question on a scale of 0 to 1. Log that score back to the run as feedback under the key 'accuracy'."

Execution Steps:

1. The agent calls list_all_lang_chain_sessions to find the ID for 'Support-Copilot'.
2. It calls list_all_lang_chain_runs_queries with a filter for the last 10 traces.
3. For each run ID returned, it calls get_single_lang_chain_run_by_id to inspect the nested inputs and outputs.
4. The agent reasons about the accuracy locally based on its system prompt.
5. It iterates through the results, calling create_a_lang_chain_feedback for each run to attach the calculated score.

Scenario 2: Golden Dataset Extraction

When a model fails in production, you want to capture that failure, correct it, and add it to a regression dataset so it never happens again.

"Find the most recent trace in the 'SQL-Generator' project that ended in an error. Extract the natural language question that caused the error, write a corrected SQL query that actually answers the question, and add this pair to the 'SQL-Correction-Dataset'."

Execution Steps:

1. The agent identifies the session ID using list_all_lang_chain_sessions.
2. It queries for the latest failed run using list_all_lang_chain_runs_queries with an error filter.
3. It fetches the detailed trace via get_single_lang_chain_run_by_id and extracts the user's natural language input.
4. The agent uses its own reasoning capabilities to write the correct SQL query.
5. It retrieves the target dataset ID via list_all_lang_chain_datasets.
6. It pushes the new, corrected Q&A pair into LangSmith using create_a_lang_chain_dataset_example.

Building Multi-Step Workflows

To build these autonomous loops, you need to programmatically bind Truto's proxy APIs to your LLM framework. The following example uses LangChain.js and the truto-langchainjs-toolset, but the architecture identical whether you use LangGraph, CrewAI, or the Vercel AI SDK.

Because Truto acts as a transparent proxy for rate limits, your agent's execution loop must handle HTTP 429 responses. The upstream LangSmith API will aggressively rate-limit bulk queries. When this happens, Truto passes the 429 status code back to you, standardized with IETF headers (ratelimit-reset). Your code must catch this, inspect the headers, sleep the thread, and resume the agent.

sequenceDiagram
    participant Agent as AI Agent
    participant Truto as Truto API Layer
    participant Upstream as LangSmith API

    Agent->>Truto: Call list_all_lang_chain_runs_queries
    Truto->>Upstream: Forward request
    Upstream-->>Truto: HTTP 429 Too Many Requests
    Truto-->>Agent: HTTP 429 with IETF headers
    Note over Agent: Parses ratelimit-reset<br>Agent sleeps for N seconds
    Agent->>Truto: Retry list_all_lang_chain_runs_queries
    Truto->>Upstream: Forward request
    Upstream-->>Truto: HTTP 200 OK
    Truto-->>Agent: Success Response

Here is how you initialize the tools and structure the agent loop to handle pagination, tool calling, and HTTP 429 backoff gracefully.

import { ChatOpenAI } from "@langchain/openai";
import { AgentExecutor, createToolCallingAgent } from "langchain/agents";
import { ChatPromptTemplate } from "@langchain/core/prompts";
import { TrutoToolManager } from "truto-langchainjs-toolset";
 
// 1. Initialize the Truto SDK with your Integrated Account ID
const toolManager = new TrutoToolManager({
  apiKey: process.env.TRUTO_API_KEY,
  integratedAccountId: process.env.LANGCHAIN_INTEGRATED_ACCOUNT_ID,
});
 
async function runAutonomousObservability() {
  // 2. Fetch the AI-ready tools for LangSmith
  const tools = await toolManager.getTools();
 
  const llm = new ChatOpenAI({
    modelName: "gpt-4o",
    temperature: 0,
  });
 
  const prompt = ChatPromptTemplate.fromMessages([
    ["system", "You are a senior AI observability engineer. You query trace data, evaluate runs, and log feedback."],
    ["placeholder", "{chat_history}"],
    ["human", "{input}"],
    ["placeholder", "{agent_scratchpad}"],
  ]);
 
  // 3. Bind the Truto tools to the agent
  const agent = createToolCallingAgent({
    llm,
    tools,
    prompt,
  });
 
  const agentExecutor = new AgentExecutor({
    agent,
    tools,
  });
 
  const userCommand = "Find the session ID for 'Production-Support', query the last 5 runs, evaluate them for accuracy, and post feedback scores.";
 
  // 4. Wrap execution in a custom retry loop to handle Truto's transparent 429s
  let success = false;
  let attempt = 0;
  const maxAttempts = 5;
 
  while (!success && attempt < maxAttempts) {
    try {
      attempt++;
      const result = await agentExecutor.invoke({
        input: userCommand,
      });
      console.log("Workflow complete:", result.output);
      success = true;
    } catch (error: any) {
      if (error?.status === 429) {
        // Read the standardized IETF header returned by Truto
        const resetTime = error.headers?.get("ratelimit-reset");
        const sleepMs = resetTime ? (parseInt(resetTime) * 1000) : (Math.pow(2, attempt) * 1000);
        
        console.warn(`Rate limit hit. Truto passed 429. Sleeping for ${sleepMs}ms`);
        await new Promise((resolve) => setTimeout(resolve, sleepMs));
      } else {
        console.error("Agent execution failed due to non-retryable error:", error);
        break;
      }
    }
  }
}
 
runAutonomousObservability();

By handling the HTTP 429 errors at the framework execution layer, you ensure that your agent does not crash during large bulk evaluation jobs, while adhering strictly to the rate limits enforced by LangSmith and passed transparently through Truto.

Moving to Production

Building AI agents that interact with external infrastructure APIs is rarely just a matter of writing a prompt. The true friction lies in the integration layer - managing auth tokens, normalizing deep JSON schema objects, and structuring deterministic function calls for the LLM.

By leveraging a unified API approach via Truto's /tools endpoint, you remove the integration boilerplate from your agent's context window. Your agent interacts with a stable, schema-enforced layer, allowing you to focus engineering cycles on evaluation logic and model reasoning rather than API maintenance.