Datastores

A datastore is a bucket, database or vector index of yours that Truto can write to. Save its credentials once in an environment, and sync jobs write their output straight into it instead of sending it to your webhook.

truto datastores list -o json

A datastore belongs to one environment. Sync jobs in that environment can write to it, and a sync job can't use another environment's datastore.

Supported types

A URL a datastore sends requests to (base_url for S3, Azure Blob Storage and Qdrant, index_host for Pinecone) must be a public https URL. Leave an optional base_url empty to use the provider's own host.

Secrets are write-only. Truto encrypts secret_access_key, credentials, sas_token, account_key, connection_string, private_key and api_key, and no response returns them. To change a host, such as base_url, bucket, region, account_name, index_host or account, send the secret again in the same request. Truto refuses to send a stored secret to a new host.

1. Create a datastore

1. Open Datastores in app.truto.one and click Create Datastore.
2. Pick a Type and fill in its fields. For S3, choose Use Amazon S3 or S3 Compatible, which asks for a Base URL.
3. Under Validation methods, tick the methods Truto should try with these credentials before it saves them.
4. Click Create Datastore.

If a validation method fails, nothing is saved, and the dialog shows each method's result. For Postgres, BigQuery, Snowflake and Pinecone the check only connects. For S3, Google Cloud Storage and Azure Blob Storage, create uploads a small test file called dev-test-file under dev/test/, so untick it if the credentials can't write there, or delete the file afterwards.

Through the API, send the same fields to POST /datastore, with validation_methods listing the checks to run. With an API token, the datastore goes into the token's environment.

truto datastores create -b '{
  "label": "Raw CRM exports",
  "type": "s3",
  "config": {
    "access_key_id": "<access key id>",
    "secret_access_key": "<secret access key>",
    "region": "us-east-1",
    "bucket": "acme-truto-exports"
  },
  "validation_methods": ["listObjects"]
}'

curl -X POST "https://api.truto.one/datastore" \
  -H "Authorization: Bearer <API token>" \
  -H "Content-Type: application/json" \
  -d '{
    "label": "Raw CRM exports",
    "type": "s3",
    "config": {
      "access_key_id": "<access key id>",
      "secret_access_key": "<secret access key>",
      "region": "us-east-1",
      "bucket": "acme-truto-exports"
    },
    "validation_methods": ["listObjects"]
  }'

When validation fails, the API returns 400 with "message": "Validation failed" and each failed method's result under data.

2. Test it

On Datastores, open a datastore's menu and choose Send test method. Pick the method and, if you want, send a custom body. Through the API, call POST /datastore/{id}/test/{method} with the method's input as the JSON body. The method must be one of the type's own, from the table above. S3, Google Cloud Storage, Azure Blob Storage, Postgres, BigQuery, Snowflake and Pinecone can be tested. Qdrant and MongoDB Data API can't.

# Lists objects (S3, Google Cloud Storage and Azure Blob Storage)
truto datastores test <datastore_id>

curl -X POST "https://api.truto.one/datastore/<datastore_id>/test/listObjects" \
  -H "Authorization: Bearer <API token>" \
  -H "Content-Type: application/json" \
  -d '{"prefix": "exports/"}'

3. Write sync job output to it

A runtime version 4 sync job writes to a datastore from a destination node with destination_type: "datastore". The node names the datastore in config.id, the method to call in method, and the method's input in config.config. Pass the datastore's ID in the run's args through an args_schema entry with format: "datastore_id", and Truto checks that it belongs to the job's environment before the run starts.

{
  "name": "upload-to-s3",
  "type": "destination",
  "destination_type": "datastore",
  "method": "uploadObject",
  "depends_on": "events-to-json",
  "config": {
    "id": "{{args.s3_datastore_id}}",
    "config": {
      "path": "crm/contacts/{{args.integrated_account_id}}",
      "file_name": "{{payload.records.0.file_name}}",
      "content": "{{payload.records.0.content}}",
      "headers": { "Content-Type": "application/json" }
    }
  },
  "resources_to_persist": ["events-to-json"]
}

What each method takes:

A node that calls a method its datastore type doesn't have fails. Runtime versions 2 and 3 have no destination nodes: a run's datastore_id sends its records to a MongoDB Data API datastore, one collection per resource. See Creating a RapidBridge sync job.

Next

The full set of datastore endpoints is in the Datastores API reference. For the rest of a V4 job, such as transforms that turn records into Parquet before an upload, see the V4 sync job reference.