> ## Documentation Index
> Fetch the complete documentation index at: https://docs.elementum.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Trace Forwarding Troubleshooting

> Fix missing spans and broken trace chains, and review what trace forwarding sends out of Elementum.

## No spans arrive

Work through these in order.

1. **Test the endpoint.** In ![Settings icon](https://mintcdn.com/elementum/TFCVHNVI8zhq54sg/images/icons/settings.svg?fit=max\&auto=format\&n=TFCVHNVI8zhq54sg\&q=85\&s=3ffc43e8a0875412cc27335241aeb4c8) **Organization Settings** > **Platform** > **Trace Forwarding**, select **Test Connection** on the endpoint. If it fails, fix the URL or credentials first. For HTTP, the URL must include the traces path, for example `https://otlp.example.com:4318/v1/traces`.
2. **Find the test span.** Search your backend for a span named `elementum.connection-test`. If the test says **Message delivered** but you cannot find the span, your backend accepted it and stored it somewhere else. Check the dataset, index, tenant, or project the credentials point to.
3. **Check the endpoint type.** Automation spans go only to **General** endpoints, never to GenAI endpoints. A GenAI endpoint receives spans only when an AI agent runs.
4. **Check that the endpoint is enabled.** A disabled endpoint receives nothing.
5. **Check the environment.** Open ![Settings icon](https://mintcdn.com/elementum/TFCVHNVI8zhq54sg/images/icons/settings.svg?fit=max\&auto=format\&n=TFCVHNVI8zhq54sg\&q=85\&s=3ffc43e8a0875412cc27335241aeb4c8) **Organization Settings** > **Platform** > **Environments**, select the environment the automation runs in, and confirm the endpoint is listed under **Trace Forwarding**. Endpoints are attached per environment.
6. **Wait a minute.** Configuration changes can take about a minute to apply. Spans usually arrive within seconds after that.
7. **Run something that produces spans.** Trigger an automation and look for `service.name = elementum`. Spans are sent as each step ends, so a run that is still in progress may show only some spans.
8. **Check your endpoint's response time.** Elementum waits up to 5 seconds for each batch. If your collector or backend is slower than that, batches are dropped. Put an OpenTelemetry Collector in front of a slow backend if you need more time to accept each batch.

Spans that were dropped while an endpoint was failing, slow, or disabled are not resent.

## Some spans are missing from a run

| Symptom | Likely reason |
| - | - |
| A step you built has no span | It is a structural step, such as a parallel branch. See [Actions without a span](https://docs.elementum.io/administration/trace-forwarding/span-reference.md#actions-without-a-span). |
| A sub-automation has no trigger or workflow span | It ran inside the calling run. Its action spans are under the calling run's workflow span. |
| No `GET` or `POST` span under `send_api_request` | The HTTP client span has not reached your environment yet. See the availability note in [Send trace headers to other systems](https://docs.elementum.io/administration/trace-forwarding/send-trace-headers.md). |
| Spans of a run arrived partly | One batch failed to deliver. Elementum does not resend it. |
| The latest steps are missing | The run is still in progress, or its spans are on their way. |

## The Elementum spans are not attached to my span

You sent `traceparent` on a call to Elementum, but the Elementum spans are a separate trace or a separate root.

* **Use a webhook trigger.** Only webhook triggers continue a caller's trace. Other APIs and trigger types start their own trace.
* **Check the header.** It must be a valid W3C `traceparent`: `00-<32 lower-case hex>-<16 lower-case hex>-<2 hex>`, with a trace id and span id that are not all zeros. Invalid headers are ignored, and the webhook is not rejected.
* **Check that the parent span exists.** The trigger's parent is the span id in your header. If your application did not export that span, your backend shows the trigger with a missing parent.
* **Check your sampling.** If your application sampled its own span out, your backend has only Elementum's spans. Elementum always sends its spans, whatever the sampled flag says.

## The workflow span is not under the trigger span

For runs that did not continue your trace, the workflow span can arrive without a parent, as a second top-level span in the trigger's trace. This is current behavior. Its span link points to the trigger, and every action span carries the run id, so the run is still complete. See [Parenting of the workflow span](https://docs.elementum.io/administration/trace-forwarding/span-reference.md#parenting-of-the-workflow-span).

## The partner system's spans are not under the Elementum call

* Turn on **Include Trace Header** on that Send API Request step and publish the workflow.
* Confirm the partner reads `traceparent`. Check the request it received.
* A redirect to a different host, scheme, or port drops the header.
* If you also added trace headers on the step, the option replaces them with Elementum's `traceparent`.
* The partner must send its spans to a backend where you can see them.

## Duplicate spans

| Cause | How to tell |
| - | - |
| Two endpoints, in one or more environments, send to the same backend | The same span id appears twice. Detach one endpoint, or send to one endpoint only. |
| Retries | Several action spans with the same `elementum.automation.workflow_task_id` and increasing `elementum.automation.retry_count`. These are real attempts, not duplicates. |
| OAuth resend | Two client spans under one `send_api_request`. The second has `elementum.automation.http.resend_count = 1`. |
| One trigger started several runs | Several `automation.workflow` spans with different `elementum.automation.execution_id` values. |
| A duplicate start request | A workflow span with `elementum.automation.start_outcome = already_started`. No second run happened. |

## Organization or environment ids are missing

Elementum does not put organization or environment ids on spans. Spans are routed by organization and environment, so everything an endpoint receives comes from the environments attached to it. To label spans, add resource attributes on the endpoint, for example `deployment.environment = production`. Use one endpoint per environment if you need different labels.

## AI agent spans

| Symptom | Likely reason and what to do |
| - | - |
| No AI spans arrive at a GenAI endpoint | Work through [No spans arrive](#no-spans-arrive) first: test the endpoint, confirm it is enabled, and confirm it is attached to the environment the agent runs in. Then send a message to an agent and search for `gen_ai.operation.name = invoke_agent`. Automations do not produce spans on GenAI endpoints. |
| The test succeeds but nothing appears in the evaluation platform | The platform may file spans under a project set by headers or resource attributes. For Weave, check `wandb.entity` and `wandb.project`. See [W\&B Weave](https://docs.elementum.io/administration/trace-forwarding/evaluating-ai.md#wb-weave). |
| The evaluation platform shows spans without inputs or outputs | The platform reads only the older `gen_ai.prompt` and `gen_ai.completion` keys. Elementum sends `gen_ai.input.messages` and `gen_ai.output.messages`. See [Keys Elementum does not send](https://docs.elementum.io/administration/trace-forwarding/genai-attribute-reference.md#keys-elementum-does-not-send). |
| A `chat` span has no `gen_ai.output.messages` | The model returned no text and no tool call, or the call failed. Check the span status and `elementum.error.message`. |
| `gen_ai.input.messages` is cut off | Very long values are truncated. A long conversation history can hit the limit. |
| Token counts are missing | The provider did not report usage for that call. |
| `gen_ai.response.id` is missing | The reply was streamed. Streamed replies carry no response id. |
| An `execute_tool` span describes only one of several tools | One span covers all tool calls of one model reply. Read the full list from the `tool_calls` in the previous `chat` span. |
| Turns of one conversation are separate traces | Expected. Group them by `gen_ai.conversation.id`. |
| A called agent's turn is not under the calling agent | Expected for agents called over A2A. Each `invoke_agent` is a root on GenAI endpoints. |
| AI spans appear on a General endpoint next to automation spans | Expected. General endpoints receive AI spans too. Automation spans carry `elementum.automation.*` attributes and AI agent spans carry `gen_ai.operation.name`. Filter on either, or drop what you do not need in a collector. |
| Span names change with every model or tool | Expected. Group by `gen_ai.operation.name`, `gen_ai.request.model`, or `gen_ai.tool.name` instead of the span name. |

## Sampling

* Elementum sends every automation span it records. There is no sampling on Elementum's side for those spans, and they are always marked as sampled.
* The sampled flag in a `traceparent` you send does not change what Elementum sends.
* Sampling in your collector or backend applies to Elementum spans like any others. With tail-based sampling, keep traces that contain `service.name = elementum` spans with errors.

## Volume

Expect one trigger span per trigger firing, one workflow span per run, one span per action attempt, and one client span per Send API Request call. A busy automation on a record-update trigger can produce many spans. Use your collector to filter, for example by dropping spans with `elementum.automation.system_task = true`.

## Privacy and security

### What data leaves Elementum

The lists below are for automation spans. AI agent spans are different. They carry the full text of prompts, replies, tool arguments, and tool results, the error message text, and span events, and they go to General and GenAI endpoints alike. See [Prompt and response content](https://docs.elementum.io/administration/trace-forwarding/genai-tracing.md#prompt-and-response-content) and [Personal data on AI spans](https://docs.elementum.io/administration/trace-forwarding/genai-tracing.md#personal-data).

For automation spans, only the attributes listed in the [attribute reference](https://docs.elementum.io/administration/trace-forwarding/attribute-reference.md) are sent. In summary:

* Ids: run, automation, workflow, step, owning Object, user, and request correlation ids.
* Fixed values: trigger types, statuses, and action types.
* Names entered in Elementum: the workflow name, step names, and the owning Object's name.
* For HTTP calls: method, host, port, status code, and the URL with query values removed.
* Timings of each span.

### What is never sent on automation spans

* Record field values, comments, attachments, or email content.
* Request and response bodies, and header values, of Send API Request calls.
* Action inputs and outputs, including script code and results.
* Error messages and stack traces. A failure is described only by a fixed label or an error class name.
* Credentials. User names and passwords are removed from `url.full`, and OAuth token requests are not traced.
* Elementum's internal spans and internal trace headers.

### Points to review

* `url.full` keeps the URL path. If a Send API Request puts a secret or personal data in the path, it appears on the span. Put such values in query parameters or headers, which are not recorded.
* Step names and workflow names are sent as typed. Avoid personal data in names.
* `elementum.user_id` identifies a user in your organization by id.
* Any endpoint that receives AI agent spans, General or GenAI, stores conversation content. Check its access controls, region, and retention before you attach it to an environment where agents run.

### Transport and credentials

* Use an `https://` endpoint so spans are encrypted in transit.
* Endpoint credentials are stored by Elementum and shown only as set after saving. Rotate them by entering a new value. Use **Custom Headers** authentication for API keys, not **Additional Headers**, whose values are visible on the settings page.
* Only Organization Administrators can view or change trace forwarding.

### Turning it off

Disable the endpoint, or detach it from the environment. Elementum stops sending within about a minute. Spans already delivered stay in your backend under your retention settings.

## FAQ

> [accordion] Does tracing slow down or affect automations?
>
> No. Spans are sent in the background. A failure to record or send spans never changes the result of a run.

> [accordion] How quickly do spans arrive?
>
> Usually within seconds after each step ends.

> [accordion] Can I get spans for runs that happened before I set up forwarding?
>
> No. Only runs after the endpoint is enabled and attached are sent.

> [accordion] Why is elementum.source missing on one span?
>
> That span is the trigger of a run that continued your trace. The continued trigger span arrives without `elementum.source`. Filter on `service.name = elementum` to select every Elementum span.

> [accordion] Why does service.name say elementum for every step?
>
> All automation spans use one service name, whichever part of Elementum ran the step. Use the span name and `elementum.automation.executor` to tell steps apart. You can replace `service.name` with a resource attribute on the endpoint.

> [accordion] What should I send to Elementum support?
>
> The run id (`elementum.automation.execution_id`), the trace id, and, for a specific step, its `elementum.toe_id`.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.