--- title: "Connect Fireworks AI to Claude: Manage Datasets & Evaluators" slug: connect-fireworks-ai-to-claude-manage-datasets-evaluators date: 2026-07-07 author: Roopendra Talekar categories: ["AI & Agents"] excerpt: "Learn how to connect Fireworks AI to Claude using a managed MCP server. Automate dataset management, evaluator provisioning, and deployment scaling via AI agents." tldr: "Connect Fireworks AI to Claude using Truto's managed MCP server. Automate model evaluations, manage SFT datasets, and dynamically scale deployments without writing custom API translation layers or managing auth." canonical: https://truto.one/blog/connect-fireworks-ai-to-claude-manage-datasets-evaluators/ --- # Connect Fireworks AI to Claude: Manage Datasets & Evaluators If your team uses ChatGPT, check out our guide on [connecting Fireworks AI to ChatGPT](https://truto.one/connect-fireworks-ai-to-chatgpt-manage-models-fine-tuning/) or explore our broader integration overview on [connecting Fireworks AI to AI Agents](https://truto.one/connect-fireworks-ai-to-ai-agents-automate-jobs-deployments/). Giving a Large Language Model (LLM) read and write access to your Fireworks AI infrastructure is an engineering necessity for teams looking to automate model evaluations, dataset management, and deployment scaling. To do this securely, you need a [Model Context Protocol (MCP) server](https://truto.one/what-is-mcp-and-mcp-servers-and-how-do-they-work/). This server acts as the translation layer between Claude's tool calls and Fireworks AI's native REST APIs. You can either build and maintain this infrastructure yourself, or use a managed integration platform like Truto to dynamically generate a secure, authenticated MCP server URL. [Building a custom MCP server](https://truto.one/the-hands-on-guide-to-building-mcp-servers-for-ai-agents-2026/) means owning the entire API lifecycle. You have to handle authentication, map massive JSON schemas to MCP tool definitions, and deal with Fireworks AI's specific infrastructure APIs. Every time an endpoint updates or a deployment shape changes, you must update your server code, redeploy, and test the integration. This guide breaks down exactly how to use Truto to generate a secure, [managed MCP server](https://truto.one/managed-mcp-for-claude-full-saas-api-access-without-security-headaches/) for Fireworks AI, connect it natively to Claude, and execute complex evaluation and dataset workflows using natural language. ## The Engineering Reality of the Fireworks AI API A custom MCP server is a self-hosted integration layer. While the open MCP standard provides a predictable way for models to discover tools, the reality of implementing it against Fireworks AI is painful. You are not just integrating a simple CRUD app - you are interfacing with a complex ML infrastructure platform. If you decide to build a custom MCP server for Fireworks AI, here are the specific challenges you will face: **AIP-160 Filter Grammar and Field Masking** Fireworks AI heavily utilizes Google's API Improvement Proposals (AIPs). List endpoints do not accept simple query parameters; they require complex AIP-160 filter expressions (e.g., `state=ACTIVE AND createTime > "2024-01-01T00:00:00Z"`). Furthermore, responses return massive payloads unless you specify exact field paths using a `readMask`. If you expose these raw parameters to Claude without strict schema validation, the model will frequently hallucinate filter syntax or request invalid field masks, resulting in HTTP 400 errors. **AIP-162 Revisions and Aliases** Managing evaluators and training shapes in Fireworks AI involves AIP-162 versioning. A single evaluator can have multiple revisions, and users can assign string aliases to point to specific snapshots. Building an MCP tool to rollback an evaluator requires parsing and validating these specific identifier formats before passing them to the API. **Complex Multipart Uploads and Async Operations** Uploading a 500MB dataset or evaluator source code archive to Fireworks AI is not a single API call. It requires requesting a presigned upload URL, executing an out-of-band PUT request with specific `Content-Type` headers, and then triggering a server-side validation endpoint. Similarly, training a model or running a batch inference job returns a long-running operation (LRO) object that must be polled. **Rate Limits and 429 Errors** Fireworks AI enforces strict rate limits across its control plane and serverless pools. It is critical to understand that Truto does not retry, throttle, or apply backoff on rate limit errors. When Fireworks AI returns an HTTP 429, Truto passes that error directly to the caller. Truto normalizes upstream rate limit info into standardized headers (`ratelimit-limit`, `ratelimit-remaining`, `ratelimit-reset`) per the IETF spec. The caller - in this case, the Claude agent or your application - is responsible for implementing retry logic and backoff. Instead of building schema translation and authentication from scratch, you can use Truto. Truto derives tool definitions dynamically from documentation records, mapping Fireworks AI's endpoints into flat, LLM-friendly schemas. ## How to Generate a Fireworks AI MCP Server Truto scopes every MCP server to a single integrated account. This means the server URL contains a cryptographic token encoding the account, the authorized tools, and the expiration time. The server is entirely self-contained. You can generate this server via the Truto UI or programmatically via the API. ### Method 1: Via the Truto UI This is the fastest method for internal teams setting up Claude Desktop. 1. Log in to your Truto dashboard and navigate to the integrated account page for your Fireworks AI connection. 2. Click the **MCP Servers** tab. 3. Click **Create MCP Server**. 4. Configure your server. You can restrict the server to specific operations (like `read` only) or specific tags (like `evaluations`). 5. Copy the generated MCP server URL (e.g., `https://api.truto.one/mcp/abc123def456...`). ### Method 2: Via the Truto API For platforms provisioning AI agents programmatically, you can generate MCP servers on the fly. ```bash curl -X POST https://api.truto.one/integrated-account/{integrated_account_id}/mcp \ -H "Authorization: Bearer YOUR_TRUTO_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Fireworks AI Eval Server", "config": { "methods": ["read", "write", "custom"] }, "expires_at": "2026-12-31T23:59:59Z" }' ``` The API returns the generated configuration and the secure `url`. ```json { "id": "mcp_srv_987654321", "name": "Fireworks AI Eval Server", "config": { "methods": ["read", "write", "custom"] }, "expires_at": "2026-12-31T23:59:59Z", "url": "https://api.truto.one/mcp/a1b2c3d4e5f6g7h8..." } ``` ## How to Connect the MCP Server to Claude Once you have your Truto MCP URL, connecting it to Claude requires zero additional code. You can do this through the Claude UI for individual users or via a configuration file for automated deployments. ### Method A: Via the Claude UI If you are using Claude for Web or an enterprise workspace that supports UI-based custom connectors: 1. Open Claude and navigate to **Settings**. 2. Select **Integrations** or **Connectors** (depending on your plan level). 3. Click **Add MCP Server** or **Add custom connector**. 4. Paste the Truto MCP URL you generated earlier. 5. Click **Add**. Claude will immediately execute an `initialize` handshake and call `tools/list` to discover the Fireworks AI capabilities. ### Method B: Via Manual Configuration File If you are using Claude Desktop locally or provisioning headless agents, you configure the connection using the `claude_desktop_config.json` file. Because Truto provides a hosted URL, you use the standard Server-Sent Events (SSE) transport wrapper. Open your configuration file: * macOS: `~/Library/Application Support/Claude/claude_desktop_config.json` * Windows: `%APPDATA%\Claude\claude_desktop_config.json` Add the Fireworks AI server configuration: ```json { "mcpServers": { "fireworks_ai": { "command": "npx", "args": [ "-y", "@modelcontextprotocol/server-sse", "https://api.truto.one/mcp/a1b2c3d4e5f6g7h8..." ] } } } ``` Save the file and restart Claude Desktop. The Fireworks AI tools will now be available in your prompt context. ## Hero Tools for Fireworks AI Administration Truto exposes the entirety of the Fireworks AI API to your agents. Here are the highest-leverage tools for managing datasets, evaluators, and deployments. ### 1. List All Fireworks AI Account Datasets **Tool Name:** `list_all_fireworks_ai_account_datasets` This tool retrieves the datasets stored in your account, which are critical for supervised fine-tuning (SFT) and evaluation jobs. It supports AIP-160 filter expressions to narrow down datasets by creation time or status. > "Find all datasets in my Fireworks account created in the last 30 days that have a state of 'ACTIVE' and tell me their estimated token counts." ### 2. Create a Fireworks AI Account Evaluator V2 **Tool Name:** `create_a_fireworks_ai_account_evaluators_v_2` Evaluators score model outputs against predefined criteria. This tool provisions a custom evaluator using the Fireworks Eval Protocol, which is necessary for rigorous quality assurance on generation pipelines. > "Create a new V2 evaluator in my Fireworks account named 'code-quality-eval'. The criteria should focus on syntactical correctness and lack of hallucinations in Python outputs." ### 3. Preview Fireworks AI Account Evaluations **Tool Name:** `fireworks_ai_account_evaluations_preview` Before running a massive, costly evaluation job across a full dataset, this tool allows the agent to run a preview test using sample data. It returns summary statistics like total runtime milliseconds and sample outputs. > "Run a preview evaluation using the 'code-quality-eval' evaluator on this specific JSON sample payload. Tell me the total runtime and the resulting score." ### 4. List All Fireworks AI Account Deployments **Tool Name:** `list_all_fireworks_ai_account_deployments` Retrieves active deployments in the account. This is essential for monitoring replica counts, checking deployment status, and identifying base models in use across different clusters. > "List all active deployments in my Fireworks account. Group them by their base model and flag any deployments where the current replica count is zero." ### 5. Scale Fireworks AI Account Deployments **Tool Name:** `fireworks_ai_account_deployments_scale` Allows the agent to dynamically scale a deployment up or down. This tool is frequently used in automated capacity management workflows, allowing an agent to spin down expensive endpoints during off-hours. > "Scale the deployment 'llama-v3-8b-instruct-prod' down to exactly zero replicas to save on accelerator hour costs." ### 6. Create a Fireworks AI Account Batch Inference Job **Tool Name:** `create_a_fireworks_ai_account_batch_inference_job` Triggers an asynchronous batch inference job. The agent passes the dataset identifier and the target model, receiving a job ID in return. The agent can then use the `get_single_fireworks_ai_account_batch_inference_job_by_id` tool to poll for completion. > "Kick off a batch inference job against the 'q4-earnings-summaries' dataset using the 'mixtral-8x7b' deployment. Give me the job ID so we can track it." For the complete tool inventory and field-level schema definitions, visit the [Fireworks AI integration page](https://truto.one/integrations/detail/fireworks). ## Workflows in Action MCP servers transform LLMs from passive query engines into active system administrators. Here are two concrete workflows demonstrating how Claude orchestrates Fireworks AI operations. ### Workflow 1: Automated Evaluator Provisioning and Preview Testing Evaluating new model iterations requires creating custom scoring criteria and validating them before execution. Claude can handle this end-to-end. > "I need to evaluate our new support chatbot model. Check our existing datasets for one related to 'customer-support-q3'. If it exists, create a new V2 evaluator focused on tone and accuracy, then run a preview evaluation against it." **Execution Steps:** 1. **Query Datasets:** Claude calls `list_all_fireworks_ai_account_datasets` using an AIP-160 filter expression (`displayName:"customer-support-q3"`) to locate the dataset ID. 2. **Create Evaluator:** Claude calls `create_a_fireworks_ai_account_evaluators_v_2` passing the required JSON payload defining the evaluation criteria. 3. **Run Preview:** Claude extracts sample data from the dataset metadata and calls `fireworks_ai_account_evaluations_preview` with the new evaluator ID to ensure the scoring logic works. **Result:** The user receives a confirmation that the evaluator is built, along with the preview runtime metrics and the test scores, ensuring the logic is sound before burning compute on a full dataset. ```mermaid sequenceDiagram participant User as User participant Claude as Claude Desktop participant Truto as Truto MCP Server participant Upstream as Upstream API (Fireworks) User->>Claude: "Check datasets and create evaluator..." Claude->>Truto: tools/call (list_all_fireworks_ai_account_datasets) Truto->>Upstream: GET /v1/accounts/{id}/datasets?filter=... Upstream-->>Truto: Dataset ID Truto-->>Claude: Tool Result (Dataset Found) Claude->>Truto: tools/call (create_a_fireworks_ai_account_evaluators_v_2) Truto->>Upstream: POST /v1/accounts/{id}/evaluatorsV2 Upstream-->>Truto: Evaluator ID Truto-->>Claude: Tool Result (Created) Claude->>Truto: tools/call (fireworks_ai_account_evaluations_preview) Truto->>Upstream: POST /v1/accounts/{id}/evaluations:preview Upstream-->>Truto: Preview Result Truto-->>Claude: Tool Result (Scores) Claude-->>User: "Evaluator created. Preview score: 9.4/10." ``` ### Workflow 2: Deployment Capacity Auditing and Scaling Managing compute costs requires scaling deployments to zero when not in use. Claude can audit your infrastructure and execute scaling commands based on natural language logic. > "Audit all active deployments in our Fireworks account. If you find any deployment using H100 accelerators that hasn't processed requests recently (check the logs), scale it down to 0 replicas to save costs." **Execution Steps:** 1. **Fetch Deployments:** Claude calls `list_all_fireworks_ai_account_deployments`, noting the `acceleratorType` and `replicaCount` for each. 2. **Check Logs:** Claude calls `list_all_fireworks_ai_account_audit_logs` to review recent activity for the identified expensive deployments. 3. **Scale Infrastructure:** For idle deployments, Claude calls `fireworks_ai_account_deployments_scale` passing `replicaCount: 0`. **Result:** The user gets a detailed infrastructure report showing which deployments were analyzed, the activity levels found in the logs, and confirmation that specific H100 instances were spun down to prevent cost overruns. ## Security and Access Control Granting an LLM direct API access to machine learning infrastructure requires strict security boundaries. Truto provides four key mechanisms to restrict MCP server access: * **Method Filtering:** You can restrict a server to specific HTTP methods. Passing `methods: ["read"]` during server creation guarantees the agent can only execute `GET` or `LIST` operations, completely preventing it from scaling deployments or deleting datasets. * **Tag Filtering:** Limit the server to specific resource domains. You can configure the server to only expose tools tagged with `datasets` or `evaluators`, hiding billing and API key management endpoints entirely. * **Expiration (TTL):** Set an `expires_at` timestamp. Truto stores the token in a distributed key-value store with an automated cleanup task. Once the token expires, the server permanently drops all connections, making it ideal for temporary contractor access. * **Dual-Layer Authentication:** Enable the `require_api_token_auth` flag. This forces the Claude client to provide a valid Truto API token in the `Authorization` header on top of the unique server URL, ensuring that leaked URLs cannot be abused by external parties. ## Architect AI-Native Workflows Today Connecting Claude to Fireworks AI via a custom MCP server is an exercise in managing API drift, parsing Google AIP filters, and debugging multipart upload flows. Truto abstracts this completely, providing a documentation-driven tool generation layer that maps Fireworks AI's native capabilities into secure, executable functions. Instead of spending weeks configuring OAuth flows and mapping schemas, you can generate a secure MCP server in minutes and start automating your model evaluation pipelines immediately. > Stop writing boilerplate integration code. Start building AI workflows. > > [Talk to us](https://cal.com/truto/partner-with-truto)