> ## Documentation Index
> Fetch the complete documentation index at: https://docs.elementum.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Using Traces

> Find an automation run, find failed steps, and measure partner calls from the spans Elementum sends.

Recipes for the questions people most often ask of automation traces. Each recipe gives the idea first, then example queries for common backends. Field names and query syntax vary between backend versions and ingestion settings. Treat the queries as starting points.

## How attributes appear in common backends

| Backend | Span attribute | Resource attribute | Error status | Span kind |
| - | - | - | - | - |
| Jaeger | Tag `elementum.automation.execution_id` | Process tag | Tag `error=true` | Tag `span.kind` |
| Grafana Tempo (TraceQL) | `span.elementum.automation.execution_id` | `resource.service.name` | `status = error` | `kind = client` |
| Honeycomb | Column `elementum.automation.execution_id` | Column `service.name` | `error = true` | `span.kind` |
| Datadog | `@elementum.automation.execution_id` | `service:elementum` | `status:error` | `@span.kind` |

Jaeger displays the array attribute `elementum.automation.workflow_chain` as a JSON string.

## Find a run by its run id

Every run has an id, `elementum.automation.execution_id`, shown in the Elementum run history. It is on the workflow span and on every action span of the run.

| Backend | Query |
| - | - |
| Jaeger | Service `elementum`, tags `elementum.automation.execution_id=<run id>` |
| Tempo | `{ span.elementum.automation.execution_id = "<run id>" }` |
| Honeycomb | `WHERE elementum.automation.execution_id = <run id>` |
| Datadog | `service:elementum @elementum.automation.execution_id:<run id>` |

Open the trace from any matching span to see the whole run, including the trigger and, for a continued webhook, your own spans.

To find all runs of one automation, filter on `elementum.automation.id`. To find runs of one published version, add `elementum.automation.workflow_version`.

## Find failed steps

Failed action spans have status `ERROR` and `elementum.action.status = FAILURE`. Failed HTTP calls have status `ERROR` and an `error.type`.

| Backend | Query |
| - | - |
| Jaeger | Service `elementum`, tags `error=true` |
| Tempo | `{ resource.service.name = "elementum" && status = error }` |
| Honeycomb | `WHERE service.name = elementum AND error = true`, grouped by `elementum.automation.task_label` |
| Datadog | `service:elementum status:error` |

Read the result:

* `elementum.automation.task_label` names the step as it was named in the automation.
* `elementum.error.message` says `Executor returned FAILED` when the step completed and reported a failure, or the class name of the error when one was raised.
* A failed `send_api_request` often has a failed HTTP client span under it. Its `http.response.status_code` and `error.type` show what the remote system answered.
* `elementum.automation.retry_count` greater than `0` means Elementum retried the step. Each attempt is its own span.

For the full error detail, give the span's `elementum.toe_id` and the run id to Elementum support, or open the run in the Elementum run history.

## Measure slow partner calls

> [!NOTE]
>
> This recipe needs the HTTP client span, which is being released. See [Send trace headers to other systems](https://docs.elementum.io/administration/trace-forwarding/send-trace-headers.md).

HTTP client spans have kind `CLIENT` and carry `server.address`. Their duration covers the whole call, including the response body.

| Backend | Query |
| - | - |
| Tempo | `{ resource.service.name = "elementum" && kind = client && duration > 2s }` |
| Tempo metrics | `{ resource.service.name = "elementum" && kind = client } \| quantile_over_time(duration, .95) by (span.server.address)` |
| Honeycomb | `WHERE service.name = elementum AND span.kind = client`, visualize `P95(duration_ms)`, group by `server.address` |
| Datadog | `service:elementum @span.kind:client @duration:>2s`, then group by `@server.address` |
| Jaeger | Service `elementum`, tags `span.kind=client`, minimum duration `2s` |

* Group by `server.address` to compare partners, or by `elementum.automation.task_label` on the parent span to compare steps.
* `url.full` keeps the path, so you can also group by endpoint. Query values are always `REDACTED`.
* Error rate per partner: count client spans with `error.type` set, grouped by `server.address`.

## Follow a trace into a partner system

1. Turn on **Include Trace Header** on the Send API Request step. See [Send trace headers to other systems](https://docs.elementum.io/administration/trace-forwarding/send-trace-headers.md).
2. Make sure the partner system reads `traceparent` and sends its spans to a backend you can query. If it uses a different backend, share the trace id with the partner. It is the same in both places.
3. Open the run's trace. The partner's server span is a child of the Elementum client span.

If the partner span is missing, check whether the call was redirected to another host. No header is sent after such a redirect. Also check whether the partner sampled the request out.

## See a run from your own request

Send `traceparent` on your webhook call. See [Continue your trace into Elementum](https://docs.elementum.io/administration/trace-forwarding/continue-your-trace.md). Then search for your own request as usual. The Elementum spans are under your client span.

## Measure how long a run took

The `automation.workflow` span marks the start of a run and ends quickly, so its duration is not the run's duration. The exception is a run started by a synchronous execution request, where the workflow span covers the whole run. Otherwise:

* Measure from the start of `automation.trigger` to the end of the last action span of the run.
* Or group action spans by `elementum.automation.execution_id` and take the earliest start and latest end.

A batched action's span covers its whole batch, so a run whose last step was batched can measure slightly longer than the step itself took. See [Batched actions](https://docs.elementum.io/administration/trace-forwarding/span-reference.md#batched-actions).

## Find runs that one trigger started

One trigger firing can start several runs. They are all in the trigger's trace.

* For a continued webhook, the workflow spans are children of the trigger span. In Tempo: `{ name = "automation.trigger" && span.elementum.automation.trigger_type = "WEB_HOOK" } >> { name = "automation.workflow" }`
* For other runs, each workflow span has a span link to the trigger span and may not be its child. Open the trigger's trace and list its `automation.workflow` spans, or follow the links if your backend shows them.

## Track automations in dashboards

Useful groupings for dashboards and alerts:

| Question | Group or filter by |
| - | - |
| Runs per automation | Count of `automation.workflow` spans by `elementum.automation.id` |
| Failure rate per step | Action spans by `elementum.automation.task_label`, ratio with `error = true` |
| Which triggers fire most | `automation.trigger` spans by `elementum.automation.trigger_type` |
| Slowest steps | Action span duration by `elementum.automation.executor` or `task_label` |
| Retries | Action spans with `elementum.automation.retry_count > 0` |
| Effect of a new version | Any of the above, by `elementum.automation.workflow_version` |

Exclude Elementum's bookkeeping steps with `elementum.automation.system_task != true`.

Because delivery is best effort, use these for trends and alerts, not for billing or audit counts.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.