Connect Z.ai to AI Agents: Automate AI Visuals and Translations

You want to connect Z.ai to an AI agent so your system can independently generate video from text, transcribe complex audio, translate documents with glossaries, and extract text from complex visual layouts. Here is exactly how to do it using Truto's /tools endpoint and SDK, bypassing the need to write custom polling loops or maintain complex multimodal API wrappers.

Giving a Large Language Model (LLM) read and write access to a heavy media generation platform like Z.ai is an engineering challenge. You either spend weeks building a custom connector that understands the difference between synchronous endpoints and long-running asynchronous background tasks, or you use a managed infrastructure layer that handles the API schema translation for you. If your team uses ChatGPT, check out our guide on connecting Z.ai to ChatGPT, or if you are building on Anthropic's models, read our guide on connecting Z.ai to Claude. For developers building custom autonomous workflows, you need a programmatic way to fetch these tools and bind them to your agent framework.

This guide breaks down exactly how to fetch AI-ready tools for Z.ai, bind them natively to an LLM using LangChain (or any framework like LangGraph, CrewAI, or Vercel AI SDK), and execute complex multimodal generation workflows. For a deeper look at the architecture behind this approach, refer to our research on architecting AI agents and the SaaS integration bottleneck.

The Engineering Reality of Custom Z.ai Connectors

Building AI agents is easy. Connecting them to external multimodal APIs is hard. Giving an LLM access to external generation tools sounds simple in a Jupyter Notebook. You write a fetch request and wrap it in a tool decorator. In production, this approach collapses entirely, especially with an API as diverse as Z.ai.

If you decide to build a custom Z.ai integration yourself, you own the entire API lifecycle. Z.ai introduces several highly specific integration challenges that break standard LLM assumptions.

The Asynchronous Polling Trap

Large language models operate synchronously, expecting an immediate response from a tool call to determine their next action. Z.ai's most powerful capabilities - like generating videos via CogVideoX or rendering complex GLM-Image assets - cannot execute within a standard HTTP timeout window. When you request a video generation, the API does not return a video. It returns a task_id.

If you hand-code this integration, you have to teach your agent how to handle asynchronous state. The agent must understand that receiving a task_id means it needs to wait, call a secondary status endpoint with that exact ID, evaluate the status string (PENDING, PROCESSING, SUCCESS, FAILED), and potentially sleep before trying again. LLMs are notoriously bad at writing their own polling loops without hallucinating endpoints or getting stuck in infinite recursion.

Multimodal Payload Construction

Standard REST APIs accept flat JSON objects. Z.ai's chat completion and media endpoints require deeply nested, specific multimodal arrays. A single prompt might need to contain an image URL, a text instruction, and a reference video. If the LLM generates a slightly malformed JSON structure for the messages array, the Z.ai API will reject it.

You are forced to write strict, defensive parsing layers between the LLM output and the Z.ai API to catch schema hallucinations. Maintaining these Pydantic models or Zod schemas by hand for every Z.ai endpoint drains engineering resources.

Strict Rate Limiting Economics

Video generation and layout parsing are highly compute-intensive. Z.ai enforces strict rate limits based on token usage and concurrent requests.

Truto does not retry, throttle, or apply backoff on rate limit errors. When the Z.ai upstream API returns an HTTP 429 error, Truto passes that error directly to your caller. However, Truto normalizes the upstream rate limit information into standardized IETF headers: ratelimit-limit, ratelimit-remaining, and ratelimit-reset (read more in our API rate limits documentation).

The caller (your agent framework) is entirely responsible for retry and backoff logic. Your agent loop must catch the 429 error, read the ratelimit-reset header, and halt execution until the specified timestamp. Hand-coding this intercept logic across dozens of distinct API calls is a massive source of boilerplate.

Fetching Z.ai Tools for AI Agents

Truto provides all the resources defined on an integration as tools for your LLM frameworks to use. Every integration on Truto is a comprehensive JSON object mapping the underlying product's API to standard REST methods. Truto handles the authentication injection and query parameter processing, exposing these as Proxy APIs.

To give your AI agent access to Z.ai, you simply call the /integrated-account/<id>/tools endpoint. This returns an array of structured JSON schema definitions for every available Z.ai method, perfectly formatted for LLM tool binding.

curl -X GET "https://api.truto.one/integrated-account/YOUR_ACCOUNT_ID/tools" \
  -H "Authorization: Bearer YOUR_TRUTO_API_KEY"

The response contains schemas ready to be passed into .bindTools() in LangChain or your framework of choice. As Z.ai updates their endpoints, Truto's schemas update automatically.

Z.ai Hero Tools for AI Agents

Rather than dumping the entire Z.ai API surface area onto your agent, Truto allows you to expose specific, high-leverage proxy methods. Here are the core "hero tools" that enable complex multimodal workflows.

Create a Video Generation Task

The create_a_z_ai_videos_generation tool initiates an asynchronous video generation task using CogVideoX or Vidu models. The agent can provide a text prompt, an image URL, or a pair of first/last frame images. It returns the tracking id and task_status.

Usage note: Because this is an async task, the agent must be instructed to retain the returned id to check the status later.

"Take this prompt: 'A futuristic city skyline at sunset with flying cars' and generate a high-quality video using the Vidu model. Give me the task ID when it starts."

Get Async Result by ID

The get_single_z_ai_paas_async_result_by_id tool is the counterpart to media generation. The agent passes a specific id to retrieve the current status of the job. If successful, it returns the AsyncVideoGenerationResponse or AsyncImageGenerationResponse containing the final asset URLs.

Usage note: Teach your agent to handle PROCESSING states gracefully by using a system prompt that dictates a waiting period before retrying.

"Check the status of task ID 'vid_987654321'. If it is finished, return the final video URL. If it is still processing, let me know."

Create a Layout Parsing Task

The create_a_z_ai_paas_layout_parsing tool utilizes the GLM-OCR model to extract structured text and layout data from complex document images or PDFs. This is critical for agents needing to read dense reports, returning markdown results, layout details, and visualization data.

Usage note: The input requires a file object. Ensure your agent has access to file reading tools to pass the correct payload format into this tool.

"Parse the layout of this financial report PDF. Extract all the text into markdown and isolate the data tables for further analysis."

Create an Audio Transcription

The create_a_z_ai_audio_transcription tool transcribes audio files into text using the GLM-ASR-2512 model. It supports multiple languages and is essential for processing meeting recordings or voice notes.

Usage note: This tool can return large text blocks. If connecting this to another downstream tool, ensure the LLM understands context window limits.

"Transcribe this user interview audio file. The file contains a mix of English and Spanish. Return the full text."

Create a General Agent Task

The create_a_z_ai_agent tool submits a task directly to Z.ai's purpose-built agent types, such as General Translation (with glossary support) or GLM Slide/Poster generation. It accepts complex instructions and returns the agent's specific completion data.

Usage note: Use this when you want to offload a highly specialized task (like generating a slide deck layout from natural language) to Z.ai's internal orchestrator rather than prompting your own LLM to do it.

"Create a slide generation agent task. The presentation should be 5 slides about Q3 revenue growth, using a professional blue color scheme."

Create a Chat Completion

The create_a_z_ai_chat_completion tool gives your workflow access to Z.ai's core conversational models. It supports multimodal inputs (text, image, video, file) and function calling.

Usage note: This is highly useful for sub-agent delegation. Your main orchestrator agent can spin off a task to a specialized Z.ai model to handle a complex reasoning step.

"Send this image of a circuit board and a text prompt asking to identify the specific microchip model to the Z.ai chat completion tool."

To view the complete inventory of available proxy endpoints, schemas, and required parameters, refer to the Z.ai integration page.

Workflows in Action

When you provide an AI agent with the right tools, it transforms from a static chatbot into an autonomous operator. Here are concrete examples of multi-step Z.ai workflows executed entirely by an agent.

Scenario 1: Automated Multilingual Video Campaign

A marketing team needs to generate promotional videos for a new product, localized for three different regions.

"Translate this English product description into French and Japanese. Then, generate a 5-second video for each language using the translated text as the prompt. Let me know when the videos are ready to download."

1. Translation Task: The agent calls create_a_z_ai_agent specifying the General Translation agent type, passing the English text and requesting French and Japanese outputs.
2. Video Generation Initiation: The agent calls create_a_z_ai_videos_generation twice (once per translated prompt), receiving two separate task_id strings.
3. Polling Loop: The agent initiates a loop, calling get_single_z_ai_paas_async_result_by_id for both IDs.
4. Completion: Once both tasks report SUCCESS, the agent parses the response payloads and returns the final video URLs to the user.

Scenario 2: Legacy Document Digitization

An operations team has scanned images of complex legal contracts that contain mixed text, signatures, and intricate table layouts that standard OCR fails to read.

"Extract the contents of this scanned contract image. Give me a structured markdown version of the document, and highlight any sections that contain pricing tables."

1. Layout Parsing: The agent calls create_a_z_ai_paas_layout_parsing with the provided image file.
2. Data Extraction: The Z.ai API returns the md_results (markdown text) and layout_details.
3. Analysis: The agent reads the parsed markdown natively within its context window, identifies the pricing tables based on the layout markers, and formats a clean response for the user containing the structured data.

Building Multi-Step Workflows

To execute these workflows reliably in production, you need an architecture that handles tool binding, execution loops, and most importantly, upstream rate limits. Truto's SDK simplifies the binding process, but your agent framework must control the retry logic.

Here is how the architecture flows when your agent application interacts with Truto and Z.ai.

sequenceDiagram
    participant Agent as "AI Agent (LangGraph/CrewAI)"
    participant TrutoSDK as "Truto SDK (TrutoToolManager)"
    participant TrutoAPI as "Truto /tools API"
    participant Zai as "Upstream API (Z.ai)"

    Agent->>TrutoSDK: Initialize ToolManager
    TrutoSDK->>TrutoAPI: GET /integrated-account/{id}/tools
    TrutoAPI-->>TrutoSDK: Return JSON Schemas
    TrutoSDK-->>Agent: LLM-ready Tools (.bindTools)
    
    Note over Agent: Agent formulates a plan
    Agent->>TrutoSDK: Execute create_a_z_ai_videos_generation
    TrutoSDK->>Zai: POST /api/paas/v4/videos/generations
    
    alt Rate Limit Exceeded
        Zai-->>TrutoSDK: 429 Too Many Requests
        TrutoSDK-->>Agent: 429 Error + ratelimit-reset header
        Note over Agent: Agent pauses execution until reset time
        Agent->>TrutoSDK: Retry execution
    end
    
    Zai-->>TrutoSDK: 200 OK (Returns task_id)
    TrutoSDK-->>Agent: task_id
    
    Note over Agent: Agent initiates polling loop
    Agent->>TrutoSDK: Execute get_single_z_ai_paas_async_result_by_id
    TrutoSDK->>Zai: GET /api/paas/v4/async-result/{id}
    Zai-->>TrutoSDK: Status: SUCCESS + Asset URLs
    TrutoSDK-->>Agent: Final Data

Below is a conceptual example using LangChain.js and the Truto SDK (truto-langchainjs-toolset). This script fetches the tools, binds them to the model, and includes a wrapper to gracefully handle Truto's standard ratelimit-reset headers when the Z.ai API enforces limits.

import { ChatOpenAI } from "@langchain/openai";
import { TrutoToolManager } from "@trutohq/truto-langchainjs-toolset";
import { AgentExecutor, createToolCallingAgent } from "langchain/agents";
import { ChatPromptTemplate } from "@langchain/core/prompts";
 
async function runZaiAgent() {
  // 1. Initialize the Truto Tool Manager with your integrated account ID
  const toolManager = new TrutoToolManager({
    apiKey: process.env.TRUTO_API_KEY,
    integratedAccountId: "YOUR_ZAI_INTEGRATED_ACCOUNT_ID"
  });
 
  // 2. Fetch the Z.ai tools
  const tools = await toolManager.getTools();
  
  // 3. Initialize the LLM
  const llm = new ChatOpenAI({
    modelName: "gpt-4o",
    temperature: 0
  });
 
  // 4. Create the agent prompt
  const prompt = ChatPromptTemplate.fromMessages([
    ["system", `You are a helpful AI assistant connected to Z.ai.
    
    CRITICAL INSTRUCTIONS FOR RATE LIMITS:
    If a tool returns an HTTP 429 error, look for the 'ratelimit-reset' header in the error message.
    You must wait for the specified time before retrying the tool call. Do not fail the workflow immediately.
    
    CRITICAL INSTRUCTIONS FOR ASYNC TASKS:
    If you execute a video generation or layout parsing task, you will receive an ID.
    You must use the get_single_z_ai_paas_async_result_by_id tool to check the status.
    Wait at least 10 seconds between status checks.`],
    ["human", "{input}"],
    ["placeholder", "{agent_scratchpad}"]
  ]);
 
  // 5. Bind tools and create the executor
  const agent = createToolCallingAgent({
    llm,
    tools,
    prompt
  });
 
  const agentExecutor = new AgentExecutor({
    agent,
    tools,
    maxIterations: 15, // Allow enough iterations for polling
  });
 
  // 6. Execute a multi-step workflow
  try {
    const response = await agentExecutor.invoke({
      input: "Generate a video of a cat riding a skateboard using CogVideoX. Poll the task until it is done and give me the URL."
    });
    console.log("Agent Response:", response.output);
  } catch (error) {
    // In production, implement robust logging for 429s and failed polling states
    console.error("Workflow failed:", error);
  }
}
 
runZaiAgent();

By offloading the schema translation and API proxying to Truto, your engineering team can focus entirely on refining the agent's logic, prompt structure, and token economics, rather than maintaining manual Pydantic models for every new Z.ai capability.

Moving Past the Integration Bottleneck

Connecting AI agents to multimodal powerhouses like Z.ai shouldn't require weeks of reverse-engineering async polling mechanisms and nested JSON schemas. When you rely on hand-coded wrappers, every upstream API update from Z.ai threatens to break your agent's execution loop.

Truto's /tools endpoint fundamentally shifts this dynamic. By instantly converting Z.ai's REST methods into LLM-native tools, you bridge the gap between your agent framework and external generation capabilities. You retain total control over rate-limit handling and workflow logic while shedding the burden of infrastructure maintenance.