Connect Cohere to ChatGPT: Generate Text and Rerank Search Results

If you are building advanced Retrieval-Augmented Generation (RAG) pipelines or specialized AI workflows, you likely need to connect Cohere to ChatGPT. While OpenAI models excel at general synthesis, Cohere's purpose-built endpoints for document reranking, enterprise search, and multilingual generation are industry standards. Giving ChatGPT the ability to orchestrate Cohere's endpoints requires a Model Context Protocol (MCP) server. If your team uses Claude, check out our guide on connecting Cohere to Claude or explore our broader architectural overview on connecting Cohere to AI Agents.

You can either spend weeks writing, hosting, and maintaining a custom MCP server to translate LLM tool calls into Cohere REST API requests, or you can use a managed infrastructure layer like Truto. This guide breaks down exactly how to use Truto to dynamically generate a secure, authenticated MCP server for Cohere, connect it natively to ChatGPT, and execute complex workflows using natural language.

The Engineering Reality of the Cohere API

A custom MCP server is a self-hosted integration layer that translates an LLM's JSON-RPC tool calls into vendor-specific API requests. While the open MCP standard provides a predictable way for models to discover tools, implementing it against Cohere's API introduces specific architectural challenges.

If you decide to build a custom MCP server for Cohere, you own the entire API lifecycle. You aren't just integrating a standard CRUD database - you are orchestrating complex ML workloads with strict constraints.

Strict Batching Limits for Embeddings

When an LLM attempts to embed a list of strings for vector storage, it often tries to pass the entire document corpus at once. Cohere's v2/embed endpoint enforces a strict limit: a maximum of 96 texts per call. If an LLM passes 100 texts, the API rejects the payload. A custom MCP server must implement a chunking mechanism to intercept the tool call, split the payload into batches of 96, execute parallel requests, and stitch the multi-dimensional arrays back together before returning the result to the LLM. If you fail to do this, the LLM will hallucinate a success state or crash the session.

Dataset Type Strictness and Async Polling

Cohere provides async batch processing for massive embedding jobs or fine-tuning. This requires a multi-step sequence: uploading a dataset, initiating a job, and polling for completion. When uploading data via the /v1/datasets endpoint, the API is incredibly strict about the type parameter (e.g., it must be exactly embed-input for embedding jobs). Furthermore, LLMs are terrible at polling. If you expose an async job tool, the LLM will call the status endpoint in a tight loop, rapidly exhausting rate limits. You must explicitly document and enforce wait intervals within the MCP tool descriptions.

Rate Limits and 429 Exhaustion

Cohere enforces rate limits based on organization tiers (e.g., requests per minute and tokens per minute). When these limits are exceeded, Cohere returns an HTTP 429 Too Many Requests. Factual note on rate limits: Truto does not retry, throttle, or apply backoff on rate limit errors. When the upstream Cohere API returns a 429, Truto passes that error directly to the caller. Truto normalizes the upstream rate limit information into standardized headers (ratelimit-limit, ratelimit-remaining, ratelimit-reset) per the IETF spec. The caller (or the client running the MCP session) is entirely responsible for reading these headers and implementing backoff logic.

Creating the Cohere MCP Server

Instead of building a proxy server from scratch, you can use Truto to instantly generate a secure, authenticated MCP server URL for your connected Cohere account. Truto derives tool definitions directly from its internal integration resources and documentation records.

You can create this server using the Truto UI or programmatically via the API.

Method 1: Via the Truto UI

For quick prototyping or internal operational use, the UI is the fastest path.

1. Log into your Truto dashboard and navigate to the integrated account page for your active Cohere connection.
2. Click the MCP Servers tab.
3. Click Create MCP Server.
4. Define the configuration. You can filter the server to only allow specific methods (like create, read) or restrict tools by functional tags.
5. Copy the generated MCP server URL (e.g., https://api.truto.one/mcp/a1b2c3d4e5f6...). This URL contains a hashed cryptographic token that scopes requests securely to this specific Cohere account.

Method 2: Via the Truto API

If you are dynamically provisioning MCP access for your own users, you should automate this via the REST API. The API validates the integration, generates a secure token stored in Cloudflare KV, and returns the endpoint.

Make an authenticated POST request to /integrated-account/:id/mcp:

curl -X POST https://api.truto.one/integrated-account/{integrated_account_id}/mcp \
  -H "Authorization: Bearer YOUR_TRUTO_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Cohere RAG Worker",
    "config": {
      "methods": ["create", "list", "get"]
    },
    "expires_at": "2026-12-31T23:59:59Z"
  }'

Truto responds with the server configuration and the crucial URL:

{
  "id": "mcp_abc123",
  "name": "Cohere RAG Worker",
  "config": {
    "methods": ["create", "list", "get"]
  },
  "expires_at": "2026-12-31T23:59:59Z",
  "url": "https://api.truto.one/mcp/a1b2c3d4e5f6..."
}

Connecting the MCP Server to ChatGPT

Once you have the Truto MCP URL, you must register it as a tool provider for your LLM. ChatGPT can consume MCP endpoints natively, depending on your account tier and interface.

Method A: Via the ChatGPT UI

If you are using ChatGPT directly, you can add custom connectors via the UI. (Note: MCP support in ChatGPT is currently behind Developer Mode for Pro, Plus, Business, Enterprise, and Education accounts).

1. In ChatGPT, navigate to Settings -> Apps -> Advanced settings.
2. Toggle Developer mode to ON.
3. Under MCP servers / Custom connectors, click to add a new server.
4. Set the Name to Cohere Tools.
5. Paste the Truto MCP URL into the Server URL field.
6. Click Save. ChatGPT will immediately perform a JSON-RPC initialize handshake, discover the Cohere endpoints, and list the available tools in the sidebar.

(If you are configuring Claude Desktop instead, navigate to Settings -> Integrations -> Add MCP Server, and paste the URL.)

Method B: Via Manual Config File

If you are orchestrating an automated agent framework (like LangChain, CrewAI, or local Claude Desktop environments), you can configure the connection via a JSON file using the Server-Sent Events (SSE) transport or standard Node execution.

Add the following block to your MCP client configuration file (e.g., claude_desktop_config.json or your framework's equivalent):

{
  "mcpServers": {
    "cohere_integration": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-sse",
        "--url",
        "https://api.truto.one/mcp/a1b2c3d4e5f6..."
      ]
    }
  }
}

Hero Tools for Cohere

When ChatGPT connects to the MCP server, Truto dynamically generates tool definitions based on Cohere's API resources and documentation records. Below are the most critical, high-leverage tools available for orchestrating Cohere.

create_a_cohere_chat

This tool calls Cohere's Chat v2 endpoint to generate text responses. It requires the target model and an array of messages. It is highly optimized for complex synthesis tasks and RAG pipelines.

Usage note: You must explicitly instruct ChatGPT which Cohere model string to use (e.g., command-r-plus-08-2024 or command-r-08-2024).

"I have a transcript of a customer support call. Use the create_a_cohere_chat tool with the 'command-r-plus' model to extract all product feature requests from this transcript. Format the output as a markdown list."

create_a_cohere_rerank

This is arguably Cohere's most valuable enterprise capability. It takes a search query and a list of raw text documents, and returns an ordered array with each text assigned a semantic relevance score. This is critical for improving RAG retrieval accuracy.

Usage note: Truto automatically handles passing the document array. The API recommends up to 1,000 documents per request; longer documents are auto-truncated to the model's max_tokens_per_doc.

"I just executed a lexical keyword search for 'authentication issues' against our internal wiki and got 50 unranked articles. Pass the query and these 50 text chunks into the create_a_cohere_rerank tool using the 'rerank-english-v3.0' model to give me the top 3 most semantically relevant results."

create_a_cohere_embed

Generates text and image embeddings using the v2 Embed API. This is used to convert strings into vector representations for storage in databases like Pinecone or Milvus.

Usage note: There is a strict maximum of 96 texts per call. Image embeddings are supported only with Embed v3.0 and newer models.

"Take this list of 40 FAQ answers and use the create_a_cohere_embed tool with the 'embed-english-v3.0' model to generate embeddings for them. Return the numerical arrays so I can prep them for our vector database upload."

create_a_cohere_dataset

Uploads a file to Cohere via multipart form to create a dataset. This is required before kicking off asynchronous jobs for fine-tuning or massive batch embedding.

Usage note: The type property is strictly enforced. For embedding jobs, it must be embed-input.

"We have a JSONL file containing 50,000 historical support queries. Use the create_a_cohere_dataset tool to upload this file. Ensure you set the dataset_type exactly to 'embed-input' so we can use it for batch processing later."

create_a_cohere_embed_job

Creates an asynchronous batch embedding job. It takes an existing dataset ID and processes it entirely on Cohere's infrastructure, outputting a new dataset of type embed-output.

Usage note: Do not instruct ChatGPT to poll for job completion continuously. Have it check once, wait, or rely on a webhook fallback in your application logic.

"Using the dataset ID we just created, call the create_a_cohere_embed_job tool with the 'embed-english-v3.0' model. Provide me with the job_id so we can monitor its status over the next few hours."

create_a_cohere_classify

Classifies up to 96 text inputs by predicting the best-fitting label for each input using provided text+label example pairs as a reference.

Usage note: If you are using a fine-tuned model, you do not need to provide the examples array. If using a base model, you must provide a strong set of few-shot examples in the payload.

"Use the create_a_cohere_classify tool to categorize these 20 incoming support tickets. I have provided an array of examples mapping text to 'Billing', 'Technical', and 'Sales'. Return the classifications for the new tickets."

list_all_cohere_models

Retrieves the current list of available Cohere models, their context lengths, and their default endpoints.

Usage note: Excellent for discovery before executing complex workflows, ensuring ChatGPT knows exactly which model string to pass to the generation or embedding tools.

"Call the list_all_cohere_models tool and find the exact model ID for the latest version of Command R+. I need to ensure we use the most recent version for our text generation task."

For the complete inventory of available tools and exact JSON schema requirements, refer to the Cohere integration page.

Workflows in Action

Connecting ChatGPT to Cohere via MCP allows you to orchestrate multi-step data processing pipelines natively. Here are concrete examples of persona-specific workflows.

Scenario 1: Semantic Reranking for Knowledge Base Synthesis

A Technical Support Engineer is trying to answer a highly specific customer question about a legacy API version. The initial keyword search returns dozens of irrelevant wiki pages. The engineer wants ChatGPT to find the actual answer hidden in the noise.

"I pulled 30 documentation snippets related to 'v1 webhook signature validation' from our internal search tool. Use Cohere to rerank these snippets by relevance to the query 'How do I validate the HMAC signature on v1 webhooks?'. Take the top 2 results, and then use Cohere's Chat tool to write a clear, 3-step response for the customer based ONLY on those two snippets."

Execution Steps:

1. ChatGPT invokes create_a_cohere_rerank, passing the query and the array of 30 document strings to the rerank-english-v3.0 model.
2. Truto proxies the request to Cohere and returns the sorted array with relevance scores.
3. ChatGPT isolates the top 2 documents based on the score.
4. ChatGPT invokes create_a_cohere_chat using the command-r-plus model, passing the query and the top 2 documents as context in the messages array.
5. Cohere synthesizes the precise answer, which Truto returns to ChatGPT for presentation to the user.

flowchart TD
    User["User Prompt"] 
    Agent["ChatGPT Agent"]
    Truto["Truto MCP Server"]
    Cohere["Cohere API"]

    User -->|"Provide raw documents"| Agent
    Agent -->|"Call create_a_cohere_rerank"| Truto
    Truto -->|"POST /v1/rerank"| Cohere
    Cohere -->|"Return relevance scores"| Truto
    Truto -->|"Return sorted array"| Agent
    Agent -->|"Call create_a_cohere_chat"| Truto
    Truto -->|"POST /v2/chat"| Cohere
    Cohere -->|"Return generated summary"| Truto
    Truto -->|"Format final output"| Agent

Scenario 2: Batch Processing a Dataset for RAG

A Data Engineer needs to process 10,000 historical sales transcripts into embeddings for a new RAG application. Doing this via synchronous API calls would hit rate limits immediately.

"I have a link to a file containing 10,000 sales call summaries. Please upload this to Cohere as a dataset, ensuring the type is set to 'embed-input'. Once uploaded, kick off an async embedding job using the 'embed-english-v3.0' model, and give me the job ID."

Execution Steps:

1. ChatGPT calls create_a_cohere_dataset, attaching the file data and specifying type: 'embed-input'.
2. Cohere creates the dataset and Truto returns the dataset_id.
3. ChatGPT calls create_a_cohere_embed_job, passing the newly acquired dataset_id and specifying the embed-english-v3.0 model.
4. Cohere queues the job and Truto returns the job_id.
5. ChatGPT informs the user the job has started, providing the ID so the user can later query get_single_cohere_embed_job_by_id to check its status.

Security and Access Control

Giving an LLM direct API access to your Cohere account requires strict operational boundaries, especially considering usage costs. Truto's MCP servers are designed with built-in access controls.

• Method Filtering: When creating the server via the /integrated-account/:id/mcp endpoint, you can restrict access using the methods array. For example, setting methods: ["read"] allows the LLM to call list_all_cohere_models but prevents it from executing expensive create_a_cohere_chat or batch job operations.
• Tag Filtering: You can group tools by functional tags. If you only want the LLM to access classification endpoints, you can restrict the server to tools tagged with classification.
• Expiration Enforcement: For temporary agent access, use the expires_at property. Truto automatically stores this with an absolute expiration timestamp; once the time is reached, the server is permanently disabled and the background cleanup process wipes the token from storage.
• Dual Authentication: By default, the cryptographically hashed MCP token in the URL acts as the authentication vector. If you set require_api_token_auth: true during creation, clients connecting to the MCP server must also pass a valid Truto API token in their Authorization header. This prevents unauthorized access even if the MCP URL leaks in application logs.

Wrap-Up

Connecting ChatGPT to Cohere via an MCP server bridges the gap between OpenAI's conversational interface and Cohere's specialized, enterprise-grade ML infrastructure. Instead of forcing your engineering team to build custom JSON-RPC middleware, handle strict embedding batch logic, and write complex schemas, Truto dynamically generates the entire translation layer.

This architecture allows your developers to focus on tuning their RAG pipelines and prompts, rather than fighting with API mechanics and 429 rate limit propagation.

:::cta{buttonText="Talk to us" buttonUrl="https://cal.com/truto/partner-with-truto"} Let's architect your AI tool calling infrastructure. Talk to our engineering team today. :::