Connect Supervisely to AI Agents: Automate End-to-End Vision Ops

You want to connect Supervisely to an AI agent so your ML operations systems can independently deploy models, track video figures, orchestrate annotation jobs, and query computer vision datasets based on real-time project requirements. Here is exactly how to do it using Truto's /tools endpoint and SDK, bypassing the need to manually build and maintain complex API wrappers for your agent.

Giving a Large Language Model (LLM) read and write access to your Supervisely instance is an engineering headache. You either spend weeks building, hosting, and maintaining a custom connector that understands the deeply nested hierarchy of computer vision assets, or you use a managed infrastructure layer that handles the boilerplate for you. If your team uses ChatGPT, check out our guide on connecting Supervisely to ChatGPT, or if you are building on Anthropic's models, read our guide on connecting Supervisely to Claude. For developers building custom autonomous workflows, you need a programmatic way to fetch these tools and bind them directly to your agent framework.

This guide breaks down exactly how to fetch AI-ready tools for Supervisely, bind them natively to an LLM using LangChain (or any framework like LangGraph, CrewAI, or Vercel AI SDK), and execute complex ML orchestration workflows. For a deeper look at the architecture behind this approach, refer to our research on architecting AI agents and the SaaS integration bottleneck.

The Engineering Reality of Custom Supervisely Connectors

Building AI agents is easy. Connecting them to external SaaS APIs is hard. Giving an LLM access to external data sounds simple in a prototype. You write a Node.js function that makes a fetch request and wrap it in an @tool decorator. In production, this approach collapses entirely, especially with an ecosystem as complex as Supervisely.

If you decide to integrate Supervisely yourself, you own the entire API lifecycle. Supervisely's computer vision platform introduces several highly specific integration challenges that break standard LLM assumptions.

The Hierarchical ID Trap

Supervisely relies on a strict, deeply nested architectural hierarchy: Teams contain Workspaces, Workspaces contain Projects, Projects contain Datasets, and Datasets contain Entities (Images, Videos, Volumes, or Point Clouds). The Supervisely API is heavily ID-driven. If an LLM needs to track a bounding box in a specific video, it cannot simply pass the video name. It must resolve the Team ID, then the Workspace ID, then the Project ID, the Dataset ID, and finally the Video ID. If you hand-code this integration, you have to write complex state machines to teach the LLM how to traverse this tree. When the LLM inevitably hallucinates an ID or attempts to skip a level of the hierarchy, the API rejects the request, and the agent loop crashes.

Annotation Geometry and Modality Complexity

Supervisely is not a standard CRUD application managing text strings; it manages complex multidimensional spatial data. Annotation objects can be 2D bounding boxes, complex polygons, 3D volumetric slices, or point cloud episodes. Each of these modalities requires a highly specific JSON payload structure. For example, triggering volumetric interpolation requires a different API contract than tracking a polygon across video frames. Exposing the raw Supervisely API directly to an LLM almost guarantees hallucinated JSON schemas, resulting in malformed geometries and corrupted annotation datasets.

Asynchronous Infrastructure Management

Running inference or training a model in Supervisely requires interacting with physical compute infrastructure. The agent must first query available Supervisely Agents (nodes with specific GPU capabilities), queue a deployment task, wait for the container to initialize, and only then route inference requests to the active model. Managing this asynchronous polling state inside an LLM's context window is a nightmare.

Why a Unified Tool Layer Matters for Agent Safety

Before writing a line of integration code, decide what layer your agent talks to. This choice determines how safe your production system will be.

Direct API tools (one tool per raw Supervisely endpoint) look convenient, but they push provider quirks directly into the LLM's context. The model has to remember that a dataset move requires a srcId and destId, or that a figure update requires specific geometryType definitions. Every one of those quirks is a hallucination waiting to happen.

Truto's unified tool layer collapses these complexities. Your agent sees standardized functions with strict JSON schemas. That gives you concrete safety wins:

1. Smaller attack surface for hallucination. The LLM only ever chooses from stable function names with deterministic inputs. It never invents volumetric matrix transformations.
2. Deterministic input validation. Every tool has a strict JSON schema. Invalid arguments are rejected locally before they hit the Supervisely API, so a broken tool call fails fast instead of corrupting computer vision data.
3. Normalized Error Handling. When the upstream API throws an error, Truto normalizes it. This is especially critical for rate limiting. Truto normalizes upstream HTTP 429s into standardized IETF headers (ratelimit-limit, ratelimit-remaining, ratelimit-reset). The agent framework can reliably read these headers and execute deterministic backoff strategies without guessing the vendor's specific error payload format.

Fetching and Binding Supervisely Tools

Instead of manually wrapping the Supervisely REST API, you can programmatically fetch pre-configured, schema-validated tools using Truto's /tools endpoint. Truto exposes all underlying integration resources as Tools for your LLM frameworks.

If you are using LangChain, the truto-langchainjs-toolset handles the boilerplate of fetching these tool schemas and registering them with the model.

import { ChatOpenAI } from "@langchain/openai";
import { TrutoToolManager } from "truto-langchainjs-toolset";
 
async function runVisionAgent() {
  // 1. Initialize the LLM
  const llm = new ChatOpenAI({
    modelName: "gpt-4o",
    temperature: 0,
  });
 
  // 2. Initialize the Truto Tool Manager for your Supervisely integration
  const toolManager = new TrutoToolManager({
    trutoApiKey: process.env.TRUTO_API_KEY,
    integratedAccountId: "supervisely-acct-12345",
  });
 
  // 3. Fetch AI-ready tools from Truto's registry
  const tools = await toolManager.getTools();
  console.log(`Loaded ${tools.length} Supervisely tools`);
 
  // 4. Bind the tools natively to the LLM
  const llmWithTools = llm.bindTools(tools);
 
  // 5. Execute an autonomous vision operations request
  const response = await llmWithTools.invoke(
    "List all available GPU agents in our team, then check the status of our active annotation jobs."
  );
 
  console.log(response.tool_calls);
}
 
runVisionAgent();

If you are not using LangChain, you can call the Truto API directly to retrieve the OpenAI-compatible JSON schemas and feed them into Vercel AI SDK, CrewAI, or any custom agent loop.

curl -X GET "https://api.truto.one/integrated-account/supervisely-acct-12345/tools?methods[0]=read&methods[1]=custom" \
  -H "Authorization: Bearer YOUR_TRUTO_API_KEY"

High-Leverage Supervisely Tools for AI Agents

Exposing all 200+ Supervisely endpoints to an LLM at once will overwhelm its context window. A robust agent architecture dynamically loads only the tools required for the specific persona (e.g., MLOps vs. Data Annotation). Here are the highest-leverage tools available for automating Supervisely workflows.

List Available Supervisely Agents

Before an AI agent can deploy a model or run intense data transformation tasks, it must locate available compute infrastructure. This tool allows the LLM to query the active edge or cloud nodes attached to your Supervisely team.

Usage Notes: The LLM must provide the teamId. It can filter the results by capability type and GPU requirements to ensure the selected node has the hardware necessary for the intended task.

"Query our Supervisely team to find all available compute agents that currently have active GPUs capable of running inference tasks."

Run Deploy Task

This tool allows the LLM to autonomously deploy a trained model onto an available agent. This is the cornerstone of MLOps automation, allowing an agent to transition a model from storage to an active serving endpoint.

Usage Notes: The LLM must provide the modelId and optionally the agentId (discovered via the list agents tool) and workspaceId. It returns a taskId which the LLM can monitor.

"Take model ID 8472 and deploy it to the GPU agent we just identified in workspace ID 42. Return the task ID so we can monitor the deployment status."

Infer Deployed Model

Once a model is deployed and active, the agent can trigger inference directly on images or JSON input data. This allows the AI agent to act as a bridge, reading raw visual data, triggering a specialized computer vision model, and processing the results.

Usage Notes: The LLM provides the deployed model endpoint details and the payload. The returned payload shape depends dynamically on the model architecture (e.g., YOLO bounding boxes vs. semantic segmentation masks).

"Run inference on the newly deployed defect detection model using image ID 9931. Summarize any anomalies detected with a confidence score above 0.85."

Bulk Track Video Figures

Manual video annotation is painfully slow. This tool allows the LLM to command Supervisely's internal tracking engines to extrapolate a bounding box or polygon across a sequence of video frames automatically.

Usage Notes: Requires videoId and figureIds. The LLM can also specify tracking settings like frame count, tracker engine, and directional flow.

"Take figure ID 11204 in video ID 551 and run the tracking engine forward for the next 100 frames to auto-annotate the vehicle's trajectory."

Send AI Search Request (Embeddings)

Supervisely supports powerful embedding-based retrieval for semantic vision search. This tool allows the LLM to search a project's visual data using text prompts or reference images, effectively giving the agent multimodal RAG capabilities.

Usage Notes: Requires the projectId. The LLM can construct semantic search queries to find edge cases or specific visual features in massive datasets without manual tagging.

"Execute an AI search across project ID 88 for images containing 'nighttime glare on license plates' and return the top 10 image IDs."

Reject Job Annotations

For quality assurance workflows, the LLM can act as a supervisor, rejecting annotations in a job based on heuristic checks or secondary model validation.

Usage Notes: Requires the id of the job. The LLM can optionally specify whether to reject all annotations or only unmarked ones.

"Reject all unmarked annotations in annotation job ID 402 so they are pushed back into the queue for the human labeling team."

This is just a fraction of the operations available. For the complete inventory of entities, webhooks, point clouds, and ecosystem models, review the Supervisely integration page for exact JSON schemas and parameter requirements.

Workflows in Action

Exposing atomic tools is just the foundation. The real power of connecting Supervisely to an AI agent lies in chaining these operations to execute end-to-end ML workflows.

Scenario 1: Autonomous Model Deployment Pipeline

An MLOps engineer wants to automate the process of moving a newly trained model into production and verifying its performance.

"Find an available GPU agent in our team, deploy the newly trained YOLOv8 model (ID 773), wait for the task to start, and run inference on image ID 1024 to verify it works."

1. list_all_supervisely_agents_available: The agent queries the team for nodes with GPU capabilities, selecting the one with the lowest current load.
2. create_a_supervisely_tasks_run_deploy: The agent queues the deployment of model 773 onto the selected agent node.
3. get_single_supervisely_task_by_id: The agent polls the deployment task until the status returns as active/running.
4. supervisely_models_infer: The agent submits image 1024 to the active model, receives the bounding box payload, and confirms the deployment was successful.

Scenario 2: Data QA and Re-Annotation Orchestration

A Data Ops manager needs to handle quality control without manually clicking through hundreds of project hierarchies.

"Check project ID 99 for any annotation jobs completed today. If the job has a rejection rate higher than 10%, pause it, and create a new labeling queue for the senior review team."

1. list_all_supervisely_jobs: The agent pulls all jobs for Project 99, filtering by recent completion dates.
2. supervisely_jobs_stats: The agent inspects the statistics for a specific job, doing the math to realize the rejection rate is 14%.
3. supervisely_jobs_pause: The agent halts the active job to prevent further low-quality annotations.
4. create_a_supervisely_labeling_queue: The agent provisions a new queue containing the rejected entities and assigns the specific IDs of the senior reviewers.

Scenario 3: Multimodal RAG Data Curation

A machine learning researcher is building a specialized dataset for an edge-case model.

"Search project 205 for images matching 'foggy weather'. Create a new dataset in project 206 called 'Fog Edge Cases' and bulk move those images into it."

1. create_a_supervisely_embeddings_send_ai_search: The agent queries the semantic embedding engine of Project 205, extracting a list of image IDs that match the visual profile of fog.
2. create_a_supervisely_dataset: The agent provisions an empty dataset named 'Fog Edge Cases' in the target Project 206.
3. create_a_supervisely_images_bulk_add: The agent executes a bulk operation to attach the identified image IDs to the newly created dataset, bypassing the need to re-upload massive binary files.

Building Multi-Step Workflows

To safely execute workflows like the ones above, your AI agent architecture needs a deterministic loop (Observe, Plan, Act) to handle the back-and-forth communication with the Supervisely API.

graph TD
  Agent["Agent<br>Brain"]
  Truto["Truto<br>Tools API"]
  Upstream["Upstream API<br>(Supervisely)"]

  Agent -->|"1. Request Tools"| Truto
  Truto -->|"2. Return JSON Schemas"| Agent
  Agent -->|"3. Execute Tool Call"| Truto
  Truto -->|"4. Proxy Request"| Upstream
  Upstream -->|"5. Return 429 (Rate Limit)"| Truto
  Truto -->|"6. Forward 429 + Headers"| Agent
  Agent -->|"7. Backoff & Retry"| Truto

When chaining multiple API calls, you must plan for rate limits. Supervisely API enforces usage quotas to protect platform stability.

It is critical to understand that Truto does not automatically retry, throttle, or absorb rate limit errors. When the Supervisely API returns an HTTP 429 (Too Many Requests), Truto instantly passes that error back to your agent. However, Truto normalizes the upstream rate limit information into standardized IETF headers (ratelimit-limit, ratelimit-remaining, ratelimit-reset).

Because Truto normalizes these headers, you don't have to write vendor-specific error parsing logic. Your chosen framework (e.g., LangGraph or Vercel AI SDK) is responsible for reading the ratelimit-reset header and putting the agent to sleep before retrying the tool call. By standardizing the error contract, Truto ensures your agent fails gracefully and recovers deterministically.

Build Vision Operations Agents Faster

Connecting AI agents to Supervisely manually requires mapping complex geospatial JSON schemas, navigating nested project hierarchies, and writing endless boilerplate to manage authentication. Exposing raw endpoints to an LLM directly leads to hallucinated API calls and corrupted visual datasets.

By leveraging Truto's /tools endpoint, you abstract away the API maintenance. Your LLM gets perfectly typed, schema-validated tools that drop seamlessly into LangChain, CrewAI, or any modern agent framework. You stop writing integration code and start building autonomous machine learning operations.