# Log forwarding

> Source: https://truto.one/docs/guides/observability/log-forwarding/

Add an observability destination and Truto streams your environment's logs into the tool your team already searches: Datadog, Splunk, Microsoft Sentinel, any OpenTelemetry collector, Elasticsearch, or an S3-compatible bucket.

:::skip-reading{cli="truto observability-destinations list" skill="truto"}
:::

## Before you start

- **One environment per destination.** A destination belongs to one environment and receives only that environment's logs. Add one per environment you want to watch. An environment can have several destinations; each receives its own copy.
- **Public HTTPS only.** Every URL you enter must be a public `https` URL. Truto rejects `localhost`, private and reserved IP addresses, hostnames without a dot, and URLs with a username or password in them. Redirects are not followed.
- **Secrets are write-only.** API keys, tokens, client secrets and passwords go in a separate `secrets` object. Truto never returns them, in the console or the API.
- **Console access.** The **Observability** tab under **Settings** is shown to workspace admins.

## 1. Add a destination

### From the console

1. Open **Settings → Observability** (`app.truto.one/settings/observability`) and pick the environment.
2. Click **Add Destination**.
3. Fill in the modal:
   - **Name** — a label for the destination, for example `Production logs`.
   - **Type** — **Datadog**, **Splunk HEC**, **Microsoft Sentinel**, **OpenTelemetry (OTLP)**, **Elasticsearch**, or **Amazon S3 / GCS / R2 archive**. The type can't be changed after the destination is created.
   - The type's own fields and credentials. See [Destination reference](#destination-reference).
   - **Active** — whether the destination receives logs.
   - **Log Types** — which logs to forward. A new destination starts with only **API** checked. See [Log types](#log-types).
4. Click **Create**.

### From the API

`POST /observability-destination` creates a destination. With an API token, the destination is created in the token's environment.

| Field | Type | Meaning |
| --- | --- | --- |
| `type` | string | `datadog`, `splunk_hec`, `sentinel`, `otlp`, `elasticsearch`, or `s3`. Required. |
| `name` | string | Label for the destination. Required. |
| `config` | object | The type's settings. Required. See [Destination reference](#destination-reference). |
| `secrets` | object | The type's credentials. Write-only. Required for every type except `otlp`. |
| `is_active` | boolean | Whether the destination receives logs. Defaults to `true`. |

:::code-tabs

```bash label="cURL"
curl -X POST https://api.truto.one/observability-destination \
  -H "Authorization: Bearer <API token>" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "datadog",
    "name": "Production logs",
    "config": {
      "site": "eu",
      "tags": ["team:integrations"],
      "log_types": ["api", "durable_object"]
    },
    "secrets": { "api_key": "<DATADOG_API_KEY>" }
  }'
```

```bash label="CLI"
truto observability-destinations create \
  --type datadog \
  --name "Production logs" \
  --config '{"site":"eu","tags":["team:integrations"],"log_types":["api","durable_object"]}' \
  --secrets '{"api_key":"<DATADOG_API_KEY>"}'
```

```python label="Python"
import asyncio
from truto_python_sdk import TrutoApi

truto_api = TrutoApi(token="<API token>")

async def main():
    destination = await truto_api.observability_destinations.create({
        "type": "datadog",
        "name": "Production logs",
        "config": {
            "site": "eu",
            "tags": ["team:integrations"],
            "log_types": ["api", "durable_object"],
        },
        "secrets": {"api_key": "<DATADOG_API_KEY>"},
    })
    print(destination["id"])

asyncio.run(main())
```

:::

The response is the destination without its secrets:

```json
{
  "id": "6c1f6f0e-5b0e-4c55-9a43-2f1c3d6c8a10",
  "environment_id": "ac15abdc-b38e-47d0-97a2-69194017c199",
  "type": "datadog",
  "name": "Production logs",
  "is_active": true,
  "config": {
    "site": "eu",
    "tags": ["team:integrations"],
    "log_types": ["api", "durable_object"]
  },
  "created_at": "2026-11-02T10:12:03.000Z",
  "updated_at": "2026-11-02T10:12:03.000Z"
}
```

## 2. Test the connection

`POST /observability-destination/:id/test` sends one synthetic `api` log, with the message `truto observability destination test`, to the destination and reports what the destination answered. In the console, open the destination's menu and click **Test connection**, or click **Test connection** inside the edit modal.

To test settings before you save them, send `config` and `secrets` in the body. They replace the stored values for this one request, exactly as a save would. When you change the host of a URL that credentials are sent to, you must send `secrets` too (see [Changing a destination](#changing-a-destination)).

:::code-tabs

```bash label="cURL"
curl -X POST https://api.truto.one/observability-destination/<destination_id>/test \
  -H "Authorization: Bearer <API token>"
```

```bash label="CLI"
truto observability-destinations test <destination_id>
```

```python label="Python"
result = await truto_api.observability_destinations.test("<destination_id>")
```

:::

```json
{ "ok": true, "status": 202 }
```

On failure, `ok` is `false`, `status` is the destination's HTTP status (absent when Truto couldn't connect), and `error` explains it. Splunk, Elasticsearch and S3 failures carry the destination's own message, for example `Splunk HEC: Incorrect index (code 7)` or `S3: AccessDenied: Access Denied`.

:::callout{type="info"}
For S3, a test writes a real object to your bucket.
:::

## Destination reference

Each type takes a `config` object and a `secrets` object. Every type also accepts `config.log_types` (see [Log types](#log-types)).

### Datadog

| Field | Where | Required | Meaning |
| --- | --- | --- | --- |
| `site` | config | Yes | Your Datadog site: `us1` (datadoghq.com), `us3` (us3.datadoghq.com), `us5` (us5.datadoghq.com), `eu` (datadoghq.eu), or `ap1` (ap1.datadoghq.com). Console label: **Site / Region**. |
| `tags` | config | No | Extra tags added to every log, such as `team:integrations`. Console label: **Tags**. |
| `api_key` | secrets | Yes | A Datadog API key. Console label: **API Key**. |

Logs arrive with `service:truto-api` and the tags `log_type:<log type>` and `env:<deployment>`, plus your own tags. HTTP fields are also nested under `http` (`http.url`, `http.status_code`, `http.method`, `http.useragent`, and so on) to match Datadog's standard attributes.

### Splunk HEC

| Field | Where | Required | Meaning |
| --- | --- | --- | --- |
| `hec_url` | config | Yes | Your HTTP Event Collector URL, for example `https://http-inputs-acme.splunkcloud.com`. Truto posts to `/services/collector/event`; you can include that path or leave it off. Console label: **HEC URL**. |
| `index` | config | No | Splunk index. Leave it empty to use the token's default index. If you set it, the token must allow that index. |
| `source` | config | No | Defaults to `truto`. |
| `sourcetype` | config | No | Console label: **Source type**. Defaults to `_json`. |
| `token` | secrets | Yes | The HEC token. Console label: **HEC token**. |

Turn **indexer acknowledgment** off on the HEC token: Truto doesn't send a channel header, and Splunk answers `Data channel is missing` without one. The HEC endpoint must present a publicly trusted TLS certificate.

Each log is one HEC event: `time` (epoch seconds), `host` (`truto-api`), `source`, `sourcetype`, `index` when set, and `event`, which holds the log record.

### Microsoft Sentinel

Truto sends logs through the Azure Monitor Logs Ingestion API. In Azure you need a Log Analytics workspace with a custom table, a data collection endpoint, a data collection rule whose input stream routes to that table, and an Entra app registration with a client secret that holds the **Monitoring Metrics Publisher** role on the rule.

| Field | Where | Required | Meaning |
| --- | --- | --- | --- |
| `tenant_id` | config | Yes | Microsoft Entra tenant ID. Console label: **Tenant ID**. |
| `client_id` | config | Yes | App registration (client) ID. Console label: **Client ID**. |
| `dce_endpoint` | config | Yes | Data collection endpoint URL, for example `https://my-dce.eastus-1.ingest.monitor.azure.com`. Console label: **Data collection endpoint**. |
| `dcr_immutable_id` | config | Yes | The rule's immutable ID, for example `dcr-0123456789abcdef`. Console label: **DCR immutable ID**. |
| `stream_name` | config | Yes | The rule's input stream, for example `Custom-TrutoLogs_CL`. Console label: **Stream name**. |
| `cloud` | config | No | `public` (default), `usgov`, or `china`. Console label: **Azure cloud**. |
| `client_secret` | secrets | Yes | The app registration's client secret. Console label: **Client secret**. |

Declare these columns on the stream:

| Column | Type |
| --- | --- |
| `TimeGenerated` | datetime |
| `LogType` | string |
| `AppEnv` | string |
| `Service` | string |
| `EnvironmentId` | string |
| `IntegratedAccountId` | string |
| `Message` | string |
| `HttpMethod` | string |
| `HttpUrl` | string |
| `HttpStatusCode` | int |
| `Duration` | real |
| `RawData` | dynamic |

`RawData` holds the full log record. Azure role assignments can take several minutes to apply; until they do, ingestion answers `403` and those logs are not retried.

### OpenTelemetry (OTLP)

| Field | Where | Required | Meaning |
| --- | --- | --- | --- |
| `endpoint` | config | Yes | Your collector's OTLP/HTTP receiver, for example `https://otel.example.com`. Truto appends `/v1/logs` unless the URL already ends with it. Console label: **OTLP/HTTP endpoint**. |
| `resource_attributes` | config | No | String key-value pairs added to the resource of every log, such as `{"deployment.region": "eu"}`. They override the defaults. In the console, enter one `key=value` per line under **Resource attributes**. |
| `token` | secrets | No | Sent as `Authorization: Bearer <token>`. |
| `username`, `password` | secrets | No | Sent as basic auth. Provide both or neither. |

In the console, pick **None**, **Bearer token**, or **Basic (username and password)** under **Authentication**.

Truto sends OTLP/HTTP with JSON encoding; gRPC is not supported. The resource carries `service.name=truto-api` and `deployment.environment`. Each log is one log record: the body is the log's `message`, severity comes from the level in the message (falling back to the HTTP status: `5xx` is `ERROR`, `4xx` is `WARN`), and the remaining fields are attributes, including `log_type`. The HTTP fields are also added under their semantic-convention names: `http.request.method`, `http.response.status_code`, `url.full`, `user_agent.original`, and `client.address`.

### Elasticsearch

| Field | Where | Required | Meaning |
| --- | --- | --- | --- |
| `url` | config | Yes | Your Elasticsearch endpoint, for example `https://my-deployment.es.us-east-1.aws.elastic.cloud`. No query string or fragment. Console label: **Elasticsearch URL**. |
| `index` | config | No | Index or data stream name: lowercase, starting with a letter or digit. Defaults to the `logs-truto-default` data stream. Console label: **Index or data stream**. |
| `pipeline` | config | No | An ingest pipeline to run on every document. Console label: **Ingest pipeline**. |
| `api_key` | secrets | One of | The encoded API key from Kibana (**Stack Management → API keys**). Sent as `Authorization: ApiKey <api_key>`. |
| `username`, `password` | secrets | One of | Basic auth. Provide both. |

Truto writes with the `_bulk` API using `create` actions, which work for data streams and plain indices alike. The key needs the `create_doc` and `auto_configure` privileges on the index.

Each log is an ECS-shaped document: `@timestamp`, `message`, `log.level`, `service.name`, `service.environment`, and where present `http.request.method`, `http.response.status_code`, `url.full`, `user_agent.original`, and `client.address`. Every Truto field sits under `truto` (including `truto.log_type`), with nested values stored as JSON strings so they don't grow your mapping.

### Amazon S3, Google Cloud Storage and Cloudflare R2

| Field | Where | Required | Meaning |
| --- | --- | --- | --- |
| `bucket` | config | Yes | Bucket name. Console label: **Bucket**. |
| `region` | config | For Amazon S3 | An AWS region such as `us-east-1`. Leave it empty for GCS or R2. Console label: **Region**. |
| `endpoint` | config | For GCS and R2 | Leave it empty for Amazon S3. Use `https://storage.googleapis.com` for GCS or `https://<account_id>.r2.cloudflarestorage.com` for R2. Console label: **S3-compatible endpoint**. |
| `prefix` | config | No | Key prefix. Letters, digits and `- _ . = /` only, up to 512 characters. Console label: **Key prefix**. |
| `access_key_id` | secrets | Yes | Access key ID. For GCS, an HMAC key from **Cloud Storage → Settings → Interoperability**. |
| `secret_access_key` | secrets | Yes | Secret access key. |

The credentials only need permission to put objects: `s3:PutObject` on `arn:aws:s3:::<bucket>/<prefix>*` for AWS, or `roles/storage.objectCreator` on the bucket for GCS.

Each batch is written as one gzipped NDJSON object, partitioned by upload time in UTC:

```
<prefix>/year=YYYY/month=MM/day=DD/hour=HH/<timestamp>-<uuid>.ndjson.gz
```

Each line is one log record plus `log_type` and `app_env`. Objects are stored as `application/gzip`, and existing objects are never overwritten.

## Log types

`config.log_types` picks which logs a destination receives. Leave it out to receive all four. An empty list `[]` forwards nothing.

| Value | Console label | What it covers |
| --- | --- | --- |
| `api` | API | One record per request to the Truto API: Unified, Proxy and Custom API calls, admin API calls, MCP server calls, and webhooks that integrations send to Truto. Carries the HTTP fields, `duration`, `request_type`, and the integrated account, integration and resource involved. |
| `durable_object` | Durable Object | Long-running work tied to one record: RapidBridge sync job runs, webhook deliveries to your endpoints, processing of incoming integration webhooks, workflow runs, scheduled actions, and connection flows. The `durable_object` field names which. Sync job runs carry `sync_job_id`, `sync_job_run_id`, and `sync_job_run_event`; webhook deliveries carry `webhook_id`, `webhook_endpoint_status`, and `webhook_successful`. |
| `queue` | Queue | Background events that start sync job runs, workflows and batch jobs, and fan out webhook events. Carries `queue` and `event`. |
| `alarm` | Alarm | Scheduled timers firing: sync job and daemon job cron triggers, scheduled actions, credential refreshes, and expiring tokens. Carries `alarm_type` and `entity_id`. |

## What a forwarded log looks like

Every record has these fields. Fields with no value are left out.

| Field | Meaning |
| --- | --- |
| `timestamp` | ISO 8601 time the log was written. |
| `environment_id` | Your Truto environment. |
| `service` | Always `truto-api`. |
| `message` | One-line summary. For `api` logs: time, level (`INFO`, `WARN` for `4xx`, `ERROR` for `5xx`), client IP, request ID, method, URL and status. |
| `logs` | Log lines written while handling the event. Each string is cut to 200 characters. |
| `log_type` | `api`, `durable_object`, `queue`, or `alarm`. Added by every destination type, in the form that destination expects. |
| `app_env` | The Truto deployment that wrote the log. |

`api` records add `host`, `http_url`, `http_method`, `http_status_code`, `http_useragent`, `http_referer`, `http_version`, `http_ip`, `http_request_id`, `http_country`, the `http_url_details_*` fields, `response_headers`, `duration` (milliseconds), `request_type` (`unified`, `proxy`, `mcp`, `admin`), `integrated_account_id`, `integration`, `resource`, `method`, and `tenant_id`, among others. MCP calls also carry `mcp_server_id`, `mcp_method`, and `tool_name`.

A Unified API call, as written to S3 or inside a Splunk `event`:

```json
{
  "timestamp": "2026-11-02T10:15:42.120Z",
  "environment_id": "ac15abdc-b38e-47d0-97a2-69194017c199",
  "app_env": "production",
  "service": "truto-api",
  "message": "2026-11-02T10:15:42.120Z INFO 203.0.113.7 (8d2f9b1c4e7a1234) GET https://api.truto.one/unified/crm/contacts?integrated_account_id=0f4c1a2e-7d3b-4f8e-9a61-5c2d8e9b7f10 200",
  "host": "api.truto.one",
  "http_method": "GET",
  "http_url": "https://api.truto.one/unified/crm/contacts?integrated_account_id=0f4c1a2e-7d3b-4f8e-9a61-5c2d8e9b7f10",
  "http_status_code": 200,
  "duration": 412,
  "request_type": "unified",
  "integrated_account_id": "0f4c1a2e-7d3b-4f8e-9a61-5c2d8e9b7f10",
  "integration": "hubspot",
  "resource": "contacts",
  "log_type": "api"
}
```

Datadog, Sentinel, OTLP and Elasticsearch reshape the same fields as described under each type in [Destination reference](#destination-reference).

## Delivery

Truto sends logs in batches. When a destination answers `429`, a `5xx`, or can't be reached, the batch is retried up to five times, waiting for the `Retry-After` value or 60 seconds. Any other error, such as `400`, `401` or `403`, drops that batch. A retry can resend logs the destination already accepted, so deduplicate on your side if exact counts matter. In S3, a retry writes a new object rather than overwriting one.

To pause forwarding, turn **Active** off or send `"is_active": false`.

## Changing a destination

`PATCH /observability-destination/:id` updates a destination. In the console, click the destination's name, then **Change** next to the stored credentials to replace them.

- `config` replaces the stored config whole, so send every field the type needs. For Datadog, that means `site` even when you only change `tags`.
- `secrets` replaces the stored secrets whole. Leave it out to keep them.
- If you change the host of `hec_url`, `dce_endpoint`, the OTLP `endpoint`, or the Elasticsearch `url`, you must send `secrets` in the same request. Truto won't send stored credentials to a new host.
- Changing `type` requires both `config` and `secrets`.

`DELETE /observability-destination/:id` deletes a destination. `GET /observability-destination` lists them and `GET /observability-destination/:id` reads one.

:::code-tabs

```bash label="cURL"
curl -X PATCH https://api.truto.one/observability-destination/<destination_id> \
  -H "Authorization: Bearer <API token>" \
  -H "Content-Type: application/json" \
  -d '{ "config": { "site": "eu", "log_types": ["api"] } }'
```

```bash label="CLI"
truto observability-destinations update <destination_id> \
  --config '{"site":"eu","log_types":["api"]}'
```

```python label="Python"
await truto_api.observability_destinations.update(
    "<destination_id>",
    {"config": {"site": "eu", "log_types": ["api"]}},
)
```

:::

## Next

The same logs are searchable in the console; see [Viewing logs](/docs/guides/logs/viewing-logs). To get Slack or email alerts about API errors and API tokens instead of raw logs, set up [Notifications](/docs/guides/observability/notifications). Every endpoint is in the [observability destinations API reference](/docs/api-reference/admin/observability-destinations).