Connect Botify to AI Agents: Run BQL Queries & Large-Scale Exports

You want to connect Botify to an AI agent so your system can independently execute complex BQL queries, analyze crawl statistics, and manage large-scale URL exports based on historical context. Here is exactly how to do it using Truto's /tools endpoint and SDK, bypassing the need to manually build custom connectors or maintain complex polling logic for asynchronous API jobs.

Giving a Large Language Model (LLM) read and write access to your Botify instance is an engineering headache. You either spend weeks building, hosting, and maintaining a custom connector that understands Botify's proprietary query DSL, or you use a managed infrastructure layer that handles the boilerplate for you. If your team uses ChatGPT, check out our guide on connecting Botify to ChatGPT, or if you are building on Anthropic's models, read our guide on connecting Botify to Claude. For developers building custom autonomous workflows, you need a programmatic way to fetch these tools and bind them to your agent framework.

This guide breaks down exactly how to fetch AI-ready tools for Botify, bind them natively to an LLM using LangChain (or any framework like LangGraph, CrewAI, or Vercel AI SDK), and execute complex SEO operations. For a deeper look at the architecture behind this approach, refer to our research on architecting AI agents and the SaaS integration bottleneck.

The Engineering Reality of Custom Botify Connectors

Building AI agents is relatively straightforward with modern frameworks. Connecting them safely to external enterprise APIs is incredibly difficult. Giving an LLM access to external SEO data sounds simple in a prototype - you write a Node.js function that makes a fetch request and wrap it in a tool decorator. In production, this approach collapses, especially with an ecosystem as data-intensive as Botify.

If you decide to integrate Botify manually, you own the entire API lifecycle. Botify's API introduces several highly specific integration challenges that break standard LLM assumptions.

The BQL (Botify Query Language) Trap

Botify relies heavily on BQL for data retrieval. Unlike standard RESTful path parameters, BQL is a proprietary JSON-based DSL used to filter, sort, and aggregate massive URL datasets. When an agent needs to retrieve URLs with specific HTTP errors and a low internal PageRank, standard REST conventions fail. The agent must formulate a valid BQL JSON payload consisting of nested filters and aggs arrays.

If you hand-code this integration, you have to write complex system prompts to teach the LLM the exact syntax of BQL. When the LLM inevitably hallucinates a field name that doesn't exist in the current project's Datamodel, or nests an and operator incorrectly, the API returns a 400 Bad Request. The LLM then gets stuck in a failure loop, repeatedly trying invalid syntax.

Asynchronous Large-Scale Exports

Enterprise SEO datasets are massive. You cannot fetch 500,000 URLs in a synchronous GET request. Botify handles bulk data via asynchronous jobs. To export data, you must POST a BQL query to create a job, receive a job_id, poll the job status endpoint periodically, and eventually retrieve a temporary download URL for the resulting CSV file.

LLMs do not naturally understand asynchronous polling. If you give an agent a generic "export data" tool, it expects an immediate response containing the data. Teaching an agent to initiate a job, wait, query status, and parse a file URL requires explicit, stateful tool definitions and strict workflow orchestration.

Strict Rate Limiting (And Why Retries Belong in Your Agent Loop)

Botify enforces strict concurrency and rate limits on API requests, especially for resource-intensive BQL aggregations. When you hit these limits, Botify returns an HTTP 429 Too Many Requests status.

A critical architectural detail: Truto does not retry, throttle, or apply backoff on rate limit errors. When the upstream Botify API returns HTTP 429, Truto passes that error directly to the caller. However, Truto normalizes the upstream rate limit information into standardized headers (ratelimit-limit, ratelimit-remaining, ratelimit-reset) per the IETF specification.

Because Truto does not automatically absorb rate limit errors, your agent framework is fully responsible for handling retries and exponential backoff. This is actually a feature, not a bug, for agentic workflows. It gives your LLM orchestration layer exact control over when to pause execution, when to notify the user of a delay, or when to pivot to a different task while waiting for the rate limit window to reset.

Why a Unified Tool Layer Matters for Agent Safety

Before writing a line of integration code, you must decide what layer your agent talks to. This choice determines the safety, reliability, and accuracy of your production system.

Direct API tools - exposing raw Botify endpoints directly to the LLM - push vendor-specific quirks into the model's context window. The model has to remember that Botify requires specific headers, handles pagination via specific cursor structures, and uses BQL for queries. Every unique vendor quirk is a hallucination waiting to happen.

A unified tool layer collapses these complexities behind a standardized schema. By utilizing Truto's /integrated-account/<id>/tools endpoint, your agent sees tools like list_all_botify_urls and create_a_botify_urls_export with strict, pre-defined JSON schemas. This architectural approach provides three concrete safety wins:

1. Deterministic input validation. Every tool generated by Truto has a strict JSON schema. Invalid arguments (like a malformed BQL filter) can be rejected before they ever hit the Botify API, allowing the agent to fail fast and self-correct based on schema validation errors.
2. Smaller attack surface. The LLM only chooses from a stable list of function names. It does not need to construct raw HTTP requests or manage authentication headers.
3. Normalized Error Handling. When Botify throws a complex error, Truto standardizes it. When rate limits are hit, the agent receives a consistent HTTP 429 with standard headers, making it trivial to implement a global retry interceptor in your agent SDK.

Hero Tools for Botify Workflows

Truto provides a comprehensive proxy API layer that maps Botify's endpoints into callable resources. While Truto exposes dozens of methods, you should only bind the specific tools your agent needs for its persona.

Here are the highest-leverage hero tools for building autonomous SEO agents.

list_all_botify_analyses

This tool retrieves all crawl analyses for a specific Botify project. It returns vital metadata including the analysis slug, status, date launched, and crawl configuration.

Contextual usage: Agents use this as step one in almost every workflow to dynamically discover the analysis_slug for the most recent completed crawl, which is a required parameter for almost all subsequent data queries.

"Find the most recent completed crawl analysis for the 'acme-corp' project and tell me when it finished running."

list_all_botify_crawl_statistics

Retrieves global crawl statistics for a specific analysis. It returns high-level metrics like total URLs crawled, total HTTP errors, and average load times.

Contextual usage: This tool is perfect for triage agents. Instead of running complex BQL queries, the agent can pull top-level metrics to instantly report on the overall health of a crawl before digging deeper.

"Summarize the global crawl statistics for the latest analysis on the 'acme-corp' project. Highlight any significant spikes in 404 or 500 errors."

list_all_botify_urls

Executes a BQL query against the URLs collection to retrieve specific URL records from an analysis.

Contextual usage: This is the primary tool for detailed data extraction. The agent constructs a BQL JSON payload to filter URLs - for example, fetching all non-indexable URLs that received organic traffic.

"Fetch a list of URLs from the latest crawl that return a 404 status code but have more than 100 internal inlinks."

get_single_botify_urls_agg_by_id

Runs BQL aggregation queries against URLs in an analysis. It accepts multiple queries in the request body and returns aggregated metrics based on defined dimensions.

Contextual usage: When an agent needs to build a report rather than a raw list of URLs, this tool is essential. It allows the agent to group URLs by segment, HTTP status, or depth, and calculate metrics like average PageRank per segment.

"Run an aggregation query to show me the distribution of HTTP status codes across the 'blog' site segment for the current analysis."

create_a_botify_urls_export

Creates a new URL export job and starts a background task that compiles the BQL query results into a downloadable CSV file.

Contextual usage: Agents use this when the user requests large datasets that exceed standard pagination limits. The tool returns a job_id that the agent must track.

"I need a full export of all URLs missing meta descriptions. Start a CSV export job for this data."

get_single_botify_urls_export_by_id

Checks the status of an active CSV export job using the job_id.

Contextual usage: This tool is the second half of the export workflow. The agent calls this in a loop (with delays) until the job_status returns as completed, at which point it extracts the job_url to provide to the user.

"Check the status of export job 'job-89123'. If it is finished, give me the download link."

To view the complete inventory of available Botify tools, query schemas, and return types, visit the Botify integration page.

Workflows in Action

To understand how a unified tool layer changes agent development, let us look at how an LLM uses these tools to execute real-world SEO operations autonomously.

Scenario 1: SEO Tech Audit Triage

An SEO manager wants a quick status check on the health of their primary domain after a weekend site migration.

"Check the latest Botify crawl for the 'production-site' project. Tell me if there was an increase in 5xx errors, and if so, fetch a sample of 10 URLs experiencing this error."

Agent Execution Steps:

1. list_all_botify_analyses: The agent calls this tool with the project slug production-site to find the most recent analysis where status equals finished. It extracts the analysis_slug.
2. list_all_botify_crawl_statistics: Using the retrieved analysis_slug, the agent fetches the global stats. It parses the JSON response to look at the http_5xx metrics.
3. list_all_botify_urls: Realizing that 5xx errors have spiked, the agent formulates a BQL query filtering for http_code >= 500 and limits the result size to 10. It calls the tool and parses the returned URLs.

The user receives a concise summary of the crawl health along with a curated list of broken URLs, without ever logging into the Botify dashboard.

Scenario 2: Automated Large-Scale CSV Export

Data science teams frequently need full URL datasets mapped against Google Analytics data for custom modeling. Requesting this via chat requires managing asynchronous state.

"Generate a CSV export of all indexable URLs in the 'e-commerce' project that received zero organic visits in the last 30 days."

Agent Execution Steps:

flowchart TD
    User["User prompt"] --> Agent["LLM Agent"]
    Agent --> ListAnalyses["list_all_botify_analyses<br>(Get latest crawl slug)"]
    ListAnalyses --> Agent
    Agent --> CreateExport["create_a_botify_urls_export<br>(Initiate BQL Job)"]
    CreateExport --> Agent
    Agent --> PollExport["get_single_botify_urls_export_by_id<br>(Poll Status)"]
    PollExport --> Agent
    Agent --> Output["Return CSV Download URL"]

1. list_all_botify_analyses: The agent identifies the target analysis_slug.
2. create_a_botify_urls_export: The agent constructs a BQL payload where indexable is true and organic visits are 0. It POSTs this payload and receives a job_id (e.g., export-9942).
3. Wait/Sleep: The agent framework pauses execution.
4. get_single_botify_urls_export_by_id: The agent checks the status. If it says running, the agent waits and tries again. Once it says completed, the agent extracts the job_url.

The user receives a direct download link to a massive CSV file, entirely orchestrated by the agent.

Building Multi-Step Workflows

To build these workflows in production, you must bind Truto's tools to your LLM and implement robust error handling - specifically for Botify's rate limits.

Remember: Truto normalizes rate limits into standard headers but passes the HTTP 429 error directly to your application. Your agent loop must catch this error, read the ratelimit-reset header, and apply backoff.

Here is how to implement a rate-limit-aware agent loop using LangChain.js and the Truto SDK.

import { ChatOpenAI } from "@langchain/openai";
import { TrutoToolManager } from "truto-langchainjs-toolset";
import { AgentExecutor, createToolCallingAgent } from "langchain/agents";
import {
  ChatPromptTemplate,
  MessagesPlaceholder,
} from "@langchain/core/prompts";
 
async function runBotifyAgent(prompt: string, integratedAccountId: string) {
  // 1. Initialize the LLM
  const llm = new ChatOpenAI({
    modelName: "gpt-4-turbo-preview",
    temperature: 0,
  });
 
  // 2. Fetch Botify tools dynamically from Truto
  const toolManager = new TrutoToolManager({
    integratedAccountId: integratedAccountId,
    trutoApiKey: process.env.TRUTO_API_KEY,
  });
 
  // Fetch specific Botify tools (filtering by method type if needed)
  const tools = await toolManager.getTools();
 
  // 3. Create the Agent
  const promptTemplate = ChatPromptTemplate.fromMessages([
    ["system", "You are an expert technical SEO assistant. You manage Botify crawls and BQL queries."],
    ["human", "{input}"],
    new MessagesPlaceholder("agent_scratchpad"),
  ]);
 
  const agent = createToolCallingAgent({
    llm,
    tools,
    prompt: promptTemplate,
  });
 
  const agentExecutor = new AgentExecutor({
    agent,
    tools,
    maxIterations: 10,
  });
 
  // 4. Execute with Rate Limit Backoff
  let attempts = 0;
  const maxAttempts = 3;
 
  while (attempts < maxAttempts) {
    try {
      const result = await agentExecutor.invoke({
        input: prompt,
      });
      return result.output;
 
    } catch (error: any) {
      // Explicitly handle Truto's normalized HTTP 429 Rate Limit response
      if (error.response && error.response.status === 429) {
        attempts++;
        
        // Extract the standardized IETF rate limit header from Truto
        const resetTimeHeader = error.response.headers['ratelimit-reset'];
        const resetTimeMs = resetTimeHeader ? parseInt(resetTimeHeader, 10) * 1000 : 5000;
        
        const delay = Math.max(resetTimeMs - Date.now(), 2000); // Minimum 2s delay
        
        console.warn(`[Rate Limit Hit] Botify API limit reached. Waiting ${delay}ms before retry...`);
        await new Promise((resolve) => setTimeout(resolve, delay));
        
      } else {
        // Re-throw if it is a 400 BQL error or auth failure
        console.error("Agent execution failed:", error.message);
        throw error;
      }
    }
  }
  
  throw new Error("Max rate limit retries exceeded for Botify API.");
}
 
// Execute a test workflow
const response = await runBotifyAgent(
  "Find the latest crawl for project 'acme' and export a list of 404 URLs.",
  "botify-account-uuid"
);
console.log(response);

Why This Architecture Scales

By fetching tools dynamically via /tools, you decouple your LLM orchestration code from the underlying Botify API schema.

If Botify updates their BQL specification or adds new crawl metrics to their endpoints, you do not need to update your LangChain code, rewrite your Zod schemas, or redeploy your agent infrastructure. Truto automatically updates the tool definitions at the /tools endpoint, ensuring your agent always has the correct parameters and descriptions for its next API call.

Strategic Wrap-Up

Connecting AI agents to enterprise SEO platforms like Botify requires moving past basic API wrappers. BQL queries, asynchronous export jobs, and massive paginated datasets demand an architecture that treats external APIs as strict, validated tools rather than raw HTTP endpoints.

By using a unified tool layer, you remove the burden of managing complex data models and rate limit headers from your prompt engineering. Your agents become safer, your API calls become deterministic, and your engineering team can focus on agent reasoning instead of debugging failed BQL syntax.