Skip to main content
Recipes for the questions people most often ask of automation traces. Each recipe gives the idea first, then example queries for common backends. Field names and query syntax vary between backend versions and ingestion settings. Treat the queries as starting points.

How attributes appear in common backends

Jaeger displays the array attribute elementum.automation.workflow_chain as a JSON string.

Find a run by its run id

Every run has an id, elementum.automation.execution_id, shown in the Elementum run history. It is on the workflow span and on every action span of the run. Open the trace from any matching span to see the whole run, including the trigger and, for a continued webhook, your own spans. To find all runs of one automation, filter on elementum.automation.id. To find runs of one published version, add elementum.automation.workflow_version.

Find failed steps

Failed action spans have status ERROR and elementum.action.status = FAILURE. Failed HTTP calls have status ERROR and an error.type. Read the result:
  • elementum.automation.task_label names the step as it was named in the automation.
  • elementum.error.message says Executor returned FAILED when the step completed and reported a failure, or the class name of the error when one was raised.
  • A failed send_api_request often has a failed HTTP client span under it. Its http.response.status_code and error.type show what the remote system answered.
  • elementum.automation.retry_count greater than 0 means Elementum retried the step. Each attempt is its own span.
For the full error detail, give the span’s elementum.toe_id and the run id to Elementum support, or open the run in the Elementum run history.

Measure slow partner calls

This recipe needs the HTTP client span, which is being released. See Send trace headers to other systems.
HTTP client spans have kind CLIENT and carry server.address. Their duration covers the whole call, including the response body.
  • Group by server.address to compare partners, or by elementum.automation.task_label on the parent span to compare steps.
  • url.full keeps the path, so you can also group by endpoint. Query values are always REDACTED.
  • Error rate per partner: count client spans with error.type set, grouped by server.address.

Follow a trace into a partner system

  1. Turn on Include Trace Header on the Send API Request step. See Send trace headers to other systems.
  2. Make sure the partner system reads traceparent and sends its spans to a backend you can query. If it uses a different backend, share the trace id with the partner. It is the same in both places.
  3. Open the run’s trace. The partner’s server span is a child of the Elementum client span.
If the partner span is missing, check whether the call was redirected to another host. No header is sent after such a redirect. Also check whether the partner sampled the request out.

See a run from your own request

Send traceparent on your webhook call. See Continue your trace into Elementum. Then search for your own request as usual. The Elementum spans are under your client span.

Measure how long a run took

The automation.workflow span marks the start of a run and ends quickly, so its duration is not the run’s duration. The exception is a run started by a synchronous execution request, where the workflow span covers the whole run. Otherwise:
  • Measure from the start of automation.trigger to the end of the last action span of the run.
  • Or group action spans by elementum.automation.execution_id and take the earliest start and latest end.
A batched action’s span covers its whole batch, so a run whose last step was batched can measure slightly longer than the step itself took. See Batched actions.

Find runs that one trigger started

One trigger firing can start several runs. They are all in the trigger’s trace.
  • For a continued webhook, the workflow spans are children of the trigger span. In Tempo: { name = "automation.trigger" && span.elementum.automation.trigger_type = "WEB_HOOK" } >> { name = "automation.workflow" }
  • For other runs, each workflow span has a span link to the trigger span and may not be its child. Open the trigger’s trace and list its automation.workflow spans, or follow the links if your backend shows them.

Track automations in dashboards

Useful groupings for dashboards and alerts: Exclude Elementum’s bookkeeping steps with elementum.automation.system_task != true. Because delivery is best effort, use these for trends and alerts, not for billing or audit counts.