# Datastores

> Source: https://truto.one/docs/guides/datastores/overview/

A **datastore** is a bucket, database or vector index of yours that Truto can write to. Save its credentials once in an environment, and sync jobs write their output straight into it instead of sending it to your webhook.

:::skip-reading{cli="truto datastores list -o json" skill="truto-cli"}
:::

A datastore belongs to one environment. Sync jobs in that environment can write to it, and a sync job can't use another environment's datastore.

## Supported types

| Type | `type` | Config fields | Methods a sync job can call |
|---|---|---|---|
| S3 (or S3-compatible) | `s3` | `access_key_id`, `secret_access_key`, `region`, `bucket`, optional `base_url` | `listObjects`, `uploadObject` |
| Google Cloud Storage | `google_cloud_storage` | `bucket`, `credentials` (a service account key JSON) | `listObjects`, `uploadObject` |
| Azure Blob Storage | `azure_blob_storage` | `account_name`, `container`, `auth_type` (`sas_token` or `account_key`), the matching `sas_token` or `account_key`, optional `base_url` | `listObjects`, `uploadObject` |
| Postgres | `postgres` | `connection_string` | `testConnection`, `insertRows`, `upsertRows`, `deleteRows`, `executeStatement` |
| BigQuery | `bigquery` | `credentials` (a service account key JSON), `dataset`, optional `project_id` | `testConnection`, `insertRows` |
| Snowflake | `snowflake` | `account`, `user`, `private_key` (key-pair authentication), `warehouse`, `database`, `schema`, optional `role` | `testConnection`, `insertRows`, `executeStatement` |
| Pinecone | `pinecone` | `api_key`, `index_host` | `testConnection`, `upsert`, `query`, `deleteVectors`, `upsertDocuments`, `searchDocuments`, `deleteDocuments` |
| Qdrant | `qdrant` | `base_url`, `api_key`, `collection`, `port` (default `6333`) | `createCollection`, `upsertPoints`, `deletePoints`, `setPayload`, `search` |
| MongoDB Data API | `mongo_data_api` | `api_url`, `api_key`, `data_source`, `database` | None. Runtime versions 2 and 3 write to it with a run's `datastore_id`. |

A URL a datastore sends requests to (`base_url` for S3, Azure Blob Storage and Qdrant, `index_host` for Pinecone) must be a public `https` URL. Leave an optional `base_url` empty to use the provider's own host.

Secrets are write-only. Truto encrypts `secret_access_key`, `credentials`, `sas_token`, `account_key`, `connection_string`, `private_key` and `api_key`, and no response returns them. To change a host, such as `base_url`, `bucket`, `region`, `account_name`, `index_host` or `account`, send the secret again in the same request. Truto refuses to send a stored secret to a new host.

## 1. Create a datastore

1. Open **Datastores** in [app.truto.one](https://app.truto.one/datastore) and click **Create Datastore**.
2. Pick a **Type** and fill in its fields. For S3, choose **Use Amazon S3** or **S3 Compatible**, which asks for a **Base URL**.
3. Under **Validation methods**, tick the methods Truto should try with these credentials before it saves them.
4. Click **Create Datastore**.

If a validation method fails, nothing is saved, and the dialog shows each method's result. For Postgres, BigQuery, Snowflake and Pinecone the check only connects. For S3, Google Cloud Storage and Azure Blob Storage, **create** uploads a small test file called `dev-test-file` under `dev/test/`, so untick it if the credentials can't write there, or delete the file afterwards.

Through the API, send the same fields to `POST /datastore`, with `validation_methods` listing the checks to run. With an API token, the datastore goes into the token's environment.

:::code-tabs

```bash label="cURL"
curl -X POST "https://api.truto.one/datastore" \
  -H "Authorization: Bearer <API token>" \
  -H "Content-Type: application/json" \
  -d '{
    "label": "Raw CRM exports",
    "type": "s3",
    "config": {
      "access_key_id": "<access key id>",
      "secret_access_key": "<secret access key>",
      "region": "us-east-1",
      "bucket": "acme-truto-exports"
    },
    "validation_methods": ["listObjects"]
  }'
```

```bash label="CLI"
truto datastores create -b '{
  "label": "Raw CRM exports",
  "type": "s3",
  "config": {
    "access_key_id": "<access key id>",
    "secret_access_key": "<secret access key>",
    "region": "us-east-1",
    "bucket": "acme-truto-exports"
  },
  "validation_methods": ["listObjects"]
}'
```

:::

When validation fails, the API returns `400` with `"message": "Validation failed"` and each failed method's result under `data`.

## 2. Test it

On **Datastores**, open a datastore's menu and choose **Send test method**. Pick the method and, if you want, send a custom body. Through the API, call `POST /datastore/{id}/test/{method}` with the method's input as the JSON body. The method must be one of the type's own, from the table above. S3, Google Cloud Storage, Azure Blob Storage, Postgres, BigQuery, Snowflake and Pinecone can be tested. Qdrant and MongoDB Data API can't.

:::code-tabs

```bash label="cURL"
curl -X POST "https://api.truto.one/datastore/<datastore_id>/test/listObjects" \
  -H "Authorization: Bearer <API token>" \
  -H "Content-Type: application/json" \
  -d '{"prefix": "exports/"}'
```

```bash label="CLI"
# Lists objects (S3, Google Cloud Storage and Azure Blob Storage)
truto datastores test <datastore_id>
```

:::

## 3. Write sync job output to it

A [runtime version 4](/docs/guides/sync-jobs/v4-reference) sync job writes to a datastore from a `destination` node with `destination_type: "datastore"`. The node names the datastore in `config.id`, the method to call in `method`, and the method's input in `config.config`. Pass the datastore's ID in the run's `args` through an `args_schema` entry with `format: "datastore_id"`, and Truto checks that it belongs to the job's environment before the run starts.

```json
{
  "name": "upload-to-s3",
  "type": "destination",
  "destination_type": "datastore",
  "method": "uploadObject",
  "depends_on": "events-to-json",
  "config": {
    "id": "{{args.s3_datastore_id}}",
    "config": {
      "path": "crm/contacts/{{args.integrated_account_id}}",
      "file_name": "{{payload.records.0.file_name}}",
      "content": "{{payload.records.0.content}}",
      "headers": { "Content-Type": "application/json" }
    }
  },
  "resources_to_persist": ["events-to-json"]
}
```

What each method takes:

| Method | Input |
|---|---|
| `uploadObject` | `file_name`, `content`, optional `path` and `headers`. Google Cloud Storage also takes `upload_type` (`media` by default, or `resumable`). |
| `listObjects` | Provider list options, such as `prefix` and `delimiter`. |
| `insertRows` | `table` and `rows` (an array of objects). BigQuery also takes `insert_id_key`, `skip_invalid_rows`, `ignore_unknown_values` and `batch_size`. Snowflake takes `batch_size`. |
| `upsertRows` (Postgres) | `table`, `rows`, `conflict_columns` and optional `update_columns`. |
| `deleteRows` (Postgres) | `table` and `where`. |
| `executeStatement` | `statement`, with `params` (Postgres) or `bindings` (Snowflake). |
| `upsertDocuments` (Pinecone) | `documents`, optional `namespace` and `batch_size`. |
| `upsertPoints` (Qdrant) | `body.points`, optional `query` (`wait`, `ordering`). |

A node that calls a method its datastore type doesn't have fails. Runtime versions 2 and 3 have no destination nodes: a run's `datastore_id` sends its records to a MongoDB Data API datastore, one collection per resource. See [Creating a RapidBridge sync job](/docs/guides/rapid-bridge/creating-rapid-bridge).

## Next

The full set of datastore endpoints is in the [Datastores API reference](/docs/api-reference/admin/datastores). For the rest of a V4 job, such as transforms that turn records into Parquet before an upload, see the [V4 sync job reference](/docs/guides/sync-jobs/v4-reference).