Connect Fireworks AI to Claude: Manage Datasets & Evaluators

If your team uses ChatGPT, check out our guide on connecting Fireworks AI to ChatGPT or explore our broader integration overview on connecting Fireworks AI to AI Agents.

Giving a Large Language Model (LLM) read and write access to your Fireworks AI infrastructure is an engineering necessity for teams looking to automate model evaluations, dataset management, and deployment scaling. To do this securely, you need a Model Context Protocol (MCP) server. This server acts as the translation layer between Claude's tool calls and Fireworks AI's native REST APIs. You can either build and maintain this infrastructure yourself, or use a managed integration platform like Truto to dynamically generate a secure, authenticated MCP server URL.

Building a custom MCP server means owning the entire API lifecycle. You have to handle authentication, map massive JSON schemas to MCP tool definitions, and deal with Fireworks AI's specific infrastructure APIs. Every time an endpoint updates or a deployment shape changes, you must update your server code, redeploy, and test the integration.

This guide breaks down exactly how to use Truto to generate a secure, managed MCP server for Fireworks AI, connect it natively to Claude, and execute complex evaluation and dataset workflows using natural language.

The Engineering Reality of the Fireworks AI API

A custom MCP server is a self-hosted integration layer. While the open MCP standard provides a predictable way for models to discover tools, the reality of implementing it against Fireworks AI is painful. You are not just integrating a simple CRUD app - you are interfacing with a complex ML infrastructure platform.

If you decide to build a custom MCP server for Fireworks AI, here are the specific challenges you will face:

AIP-160 Filter Grammar and Field Masking Fireworks AI heavily utilizes Google's API Improvement Proposals (AIPs). List endpoints do not accept simple query parameters; they require complex AIP-160 filter expressions (e.g., state=ACTIVE AND createTime > "2024-01-01T00:00:00Z"). Furthermore, responses return massive payloads unless you specify exact field paths using a readMask. If you expose these raw parameters to Claude without strict schema validation, the model will frequently hallucinate filter syntax or request invalid field masks, resulting in HTTP 400 errors.

AIP-162 Revisions and Aliases Managing evaluators and training shapes in Fireworks AI involves AIP-162 versioning. A single evaluator can have multiple revisions, and users can assign string aliases to point to specific snapshots. Building an MCP tool to rollback an evaluator requires parsing and validating these specific identifier formats before passing them to the API.

Complex Multipart Uploads and Async Operations Uploading a 500MB dataset or evaluator source code archive to Fireworks AI is not a single API call. It requires requesting a presigned upload URL, executing an out-of-band PUT request with specific Content-Type headers, and then triggering a server-side validation endpoint. Similarly, training a model or running a batch inference job returns a long-running operation (LRO) object that must be polled.

Rate Limits and 429 Errors Fireworks AI enforces strict rate limits across its control plane and serverless pools. It is critical to understand that Truto does not retry, throttle, or apply backoff on rate limit errors. When Fireworks AI returns an HTTP 429, Truto passes that error directly to the caller. Truto normalizes upstream rate limit info into standardized headers (ratelimit-limit, ratelimit-remaining, ratelimit-reset) per the IETF spec. The caller - in this case, the Claude agent or your application - is responsible for implementing retry logic and backoff.

Instead of building schema translation and authentication from scratch, you can use Truto. Truto derives tool definitions dynamically from documentation records, mapping Fireworks AI's endpoints into flat, LLM-friendly schemas.

How to Generate a Fireworks AI MCP Server

Truto scopes every MCP server to a single integrated account. This means the server URL contains a cryptographic token encoding the account, the authorized tools, and the expiration time. The server is entirely self-contained.

You can generate this server via the Truto UI or programmatically via the API.

Method 1: Via the Truto UI

This is the fastest method for internal teams setting up Claude Desktop.

1. Log in to your Truto dashboard and navigate to the integrated account page for your Fireworks AI connection.
2. Click the MCP Servers tab.
3. Click Create MCP Server.
4. Configure your server. You can restrict the server to specific operations (like read only) or specific tags (like evaluations).
5. Copy the generated MCP server URL (e.g., https://api.truto.one/mcp/abc123def456...).

Method 2: Via the Truto API

For platforms provisioning AI agents programmatically, you can generate MCP servers on the fly.

curl -X POST https://api.truto.one/integrated-account/{integrated_account_id}/mcp \
  -H "Authorization: Bearer YOUR_TRUTO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Fireworks AI Eval Server",
    "config": {
      "methods": ["read", "write", "custom"]
    },
    "expires_at": "2026-12-31T23:59:59Z"
  }'

The API returns the generated configuration and the secure url.

{
  "id": "mcp_srv_987654321",
  "name": "Fireworks AI Eval Server",
  "config": { "methods": ["read", "write", "custom"] },
  "expires_at": "2026-12-31T23:59:59Z",
  "url": "https://api.truto.one/mcp/a1b2c3d4e5f6g7h8..."
}

How to Connect the MCP Server to Claude

Once you have your Truto MCP URL, connecting it to Claude requires zero additional code. You can do this through the Claude UI for individual users or via a configuration file for automated deployments.

Method A: Via the Claude UI

If you are using Claude for Web or an enterprise workspace that supports UI-based custom connectors:

1. Open Claude and navigate to Settings.
2. Select Integrations or Connectors (depending on your plan level).
3. Click Add MCP Server or Add custom connector.
4. Paste the Truto MCP URL you generated earlier.
5. Click Add. Claude will immediately execute an initialize handshake and call tools/list to discover the Fireworks AI capabilities.

Method B: Via Manual Configuration File

If you are using Claude Desktop locally or provisioning headless agents, you configure the connection using the claude_desktop_config.json file. Because Truto provides a hosted URL, you use the standard Server-Sent Events (SSE) transport wrapper.

Open your configuration file:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %APPDATA%\Claude\claude_desktop_config.json

Add the Fireworks AI server configuration:

{
  "mcpServers": {
    "fireworks_ai": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-sse",
        "https://api.truto.one/mcp/a1b2c3d4e5f6g7h8..."
      ]
    }
  }
}

Save the file and restart Claude Desktop. The Fireworks AI tools will now be available in your prompt context.

Hero Tools for Fireworks AI Administration

Truto exposes the entirety of the Fireworks AI API to your agents. Here are the highest-leverage tools for managing datasets, evaluators, and deployments.

1. List All Fireworks AI Account Datasets

Tool Name: list_all_fireworks_ai_account_datasets

This tool retrieves the datasets stored in your account, which are critical for supervised fine-tuning (SFT) and evaluation jobs. It supports AIP-160 filter expressions to narrow down datasets by creation time or status.

"Find all datasets in my Fireworks account created in the last 30 days that have a state of 'ACTIVE' and tell me their estimated token counts."

2. Create a Fireworks AI Account Evaluator V2

Tool Name: create_a_fireworks_ai_account_evaluators_v_2

Evaluators score model outputs against predefined criteria. This tool provisions a custom evaluator using the Fireworks Eval Protocol, which is necessary for rigorous quality assurance on generation pipelines.

"Create a new V2 evaluator in my Fireworks account named 'code-quality-eval'. The criteria should focus on syntactical correctness and lack of hallucinations in Python outputs."

3. Preview Fireworks AI Account Evaluations

Tool Name: fireworks_ai_account_evaluations_preview

Before running a massive, costly evaluation job across a full dataset, this tool allows the agent to run a preview test using sample data. It returns summary statistics like total runtime milliseconds and sample outputs.

"Run a preview evaluation using the 'code-quality-eval' evaluator on this specific JSON sample payload. Tell me the total runtime and the resulting score."

4. List All Fireworks AI Account Deployments

Tool Name: list_all_fireworks_ai_account_deployments

Retrieves active deployments in the account. This is essential for monitoring replica counts, checking deployment status, and identifying base models in use across different clusters.

"List all active deployments in my Fireworks account. Group them by their base model and flag any deployments where the current replica count is zero."

5. Scale Fireworks AI Account Deployments

Tool Name: fireworks_ai_account_deployments_scale

Allows the agent to dynamically scale a deployment up or down. This tool is frequently used in automated capacity management workflows, allowing an agent to spin down expensive endpoints during off-hours.

"Scale the deployment 'llama-v3-8b-instruct-prod' down to exactly zero replicas to save on accelerator hour costs."

6. Create a Fireworks AI Account Batch Inference Job

Tool Name: create_a_fireworks_ai_account_batch_inference_job

Triggers an asynchronous batch inference job. The agent passes the dataset identifier and the target model, receiving a job ID in return. The agent can then use the get_single_fireworks_ai_account_batch_inference_job_by_id tool to poll for completion.

"Kick off a batch inference job against the 'q4-earnings-summaries' dataset using the 'mixtral-8x7b' deployment. Give me the job ID so we can track it."

For the complete tool inventory and field-level schema definitions, visit the Fireworks AI integration page.

Workflows in Action

MCP servers transform LLMs from passive query engines into active system administrators. Here are two concrete workflows demonstrating how Claude orchestrates Fireworks AI operations.

Workflow 1: Automated Evaluator Provisioning and Preview Testing

Evaluating new model iterations requires creating custom scoring criteria and validating them before execution. Claude can handle this end-to-end.

"I need to evaluate our new support chatbot model. Check our existing datasets for one related to 'customer-support-q3'. If it exists, create a new V2 evaluator focused on tone and accuracy, then run a preview evaluation against it."

Execution Steps:

1. Query Datasets: Claude calls list_all_fireworks_ai_account_datasets using an AIP-160 filter expression (displayName:"customer-support-q3") to locate the dataset ID.
2. Create Evaluator: Claude calls create_a_fireworks_ai_account_evaluators_v_2 passing the required JSON payload defining the evaluation criteria.
3. Run Preview: Claude extracts sample data from the dataset metadata and calls fireworks_ai_account_evaluations_preview with the new evaluator ID to ensure the scoring logic works.

Result: The user receives a confirmation that the evaluator is built, along with the preview runtime metrics and the test scores, ensuring the logic is sound before burning compute on a full dataset.

sequenceDiagram
    participant User as User
    participant Claude as Claude Desktop
    participant Truto as Truto MCP Server
    participant Upstream as Upstream API (Fireworks)

    User->>Claude: "Check datasets and create evaluator..."
    Claude->>Truto: tools/call (list_all_fireworks_ai_account_datasets)
    Truto->>Upstream: GET /v1/accounts/{id}/datasets?filter=...
    Upstream-->>Truto: Dataset ID
    Truto-->>Claude: Tool Result (Dataset Found)
    
    Claude->>Truto: tools/call (create_a_fireworks_ai_account_evaluators_v_2)
    Truto->>Upstream: POST /v1/accounts/{id}/evaluatorsV2
    Upstream-->>Truto: Evaluator ID
    Truto-->>Claude: Tool Result (Created)
    
    Claude->>Truto: tools/call (fireworks_ai_account_evaluations_preview)
    Truto->>Upstream: POST /v1/accounts/{id}/evaluations:preview
    Upstream-->>Truto: Preview Result
    Truto-->>Claude: Tool Result (Scores)
    Claude-->>User: "Evaluator created. Preview score: 9.4/10."

Workflow 2: Deployment Capacity Auditing and Scaling

Managing compute costs requires scaling deployments to zero when not in use. Claude can audit your infrastructure and execute scaling commands based on natural language logic.

"Audit all active deployments in our Fireworks account. If you find any deployment using H100 accelerators that hasn't processed requests recently (check the logs), scale it down to 0 replicas to save costs."

Execution Steps:

1. Fetch Deployments: Claude calls list_all_fireworks_ai_account_deployments, noting the acceleratorType and replicaCount for each.
2. Check Logs: Claude calls list_all_fireworks_ai_account_audit_logs to review recent activity for the identified expensive deployments.
3. Scale Infrastructure: For idle deployments, Claude calls fireworks_ai_account_deployments_scale passing replicaCount: 0.

Result: The user gets a detailed infrastructure report showing which deployments were analyzed, the activity levels found in the logs, and confirmation that specific H100 instances were spun down to prevent cost overruns.

Security and Access Control

Granting an LLM direct API access to machine learning infrastructure requires strict security boundaries. Truto provides four key mechanisms to restrict MCP server access:

• Method Filtering: You can restrict a server to specific HTTP methods. Passing methods: ["read"] during server creation guarantees the agent can only execute GET or LIST operations, completely preventing it from scaling deployments or deleting datasets.
• Tag Filtering: Limit the server to specific resource domains. You can configure the server to only expose tools tagged with datasets or evaluators, hiding billing and API key management endpoints entirely.
• Expiration (TTL): Set an expires_at timestamp. Truto stores the token in a distributed key-value store with an automated cleanup task. Once the token expires, the server permanently drops all connections, making it ideal for temporary contractor access.
• Dual-Layer Authentication: Enable the require_api_token_auth flag. This forces the Claude client to provide a valid Truto API token in the Authorization header on top of the unique server URL, ensuring that leaked URLs cannot be abused by external parties.

Architect AI-Native Workflows Today

Connecting Claude to Fireworks AI via a custom MCP server is an exercise in managing API drift, parsing Google AIP filters, and debugging multipart upload flows. Truto abstracts this completely, providing a documentation-driven tool generation layer that maps Fireworks AI's native capabilities into secure, executable functions.

Instead of spending weeks configuring OAuth flows and mapping schemas, you can generate a secure MCP server in minutes and start automating your model evaluation pipelines immediately.