> ## Documentation Index
> Fetch the complete documentation index at: https://docs.elementum.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Trace Forwarding

> Configure OpenTelemetry endpoints to stream trace data from Elementum to your observability and AI evaluation platforms.

<Warning>
  **Beta:** The trace forwarding feature is currently being tested in beta. Not all Elementum customers have access to this functionality.
</Warning>

## Overview

Trace Forwarding lets your organization push OpenTelemetry (OTLP) trace data from Elementum environments to external platforms — observability tools, AI evaluation suites, or any system that accepts OTLP over gRPC or HTTP. You can configure multiple endpoints per channel to route traces to different teams or platforms in parallel.

Go to <img src="https://mintcdn.com/elementum/TFCVHNVI8zhq54sg/images/icons/settings.svg?fit=max&auto=format&n=TFCVHNVI8zhq54sg&q=85&s=3ffc43e8a0875412cc27335241aeb4c8" alt="Settings icon" className="inline-ui-icon" width="24" height="24" data-path="images/icons/settings.svg" /> **Organization Settings** > **Platform** > **Trace Forwarding**. Only **Organization Administrators** can view and modify these settings.

## Trace channels

Two independent channels are available:

| Channel            | What it forwards                                        | Default protocol | Example destinations                |
| ------------------ | ------------------------------------------------------- | ---------------- | ----------------------------------- |
| **General Traces** | Operational telemetry from all Elementum activities     | gRPC             | Datadog, Grafana, Jaeger, Honeycomb |
| **GenAI Traces**   | Detailed LLM and agent spans from AI-powered operations | HTTP             | Weave (W\&B), LangSmith, Arize      |

## Quick start: add your first endpoint

1. Go to <img src="https://mintcdn.com/elementum/TFCVHNVI8zhq54sg/images/icons/settings.svg?fit=max&auto=format&n=TFCVHNVI8zhq54sg&q=85&s=3ffc43e8a0875412cc27335241aeb4c8" alt="Settings icon" className="inline-ui-icon" width="24" height="24" data-path="images/icons/settings.svg" /> **Organization Settings** > **Platform** > **Trace Forwarding**.
2. Click **Add General Endpoint** or **Add GenAI Endpoint** depending on the type of traces you want to forward.
3. Enter a **Name**, your **Endpoint URL**, and confirm the **Protocol** matches what your destination platform expects.
4. Select an **Authorization Type** and fill in the required credentials.
5. Optionally, add **Resource Attributes** to tag every span with metadata such as `environment` or `team`.
6. Click **Save & Test**. A green *Message delivered* result confirms the endpoint is reachable.
7. Click **Enable** to activate the endpoint.
8. Go to **Organization Settings** > **Platform** > **Environments**, click **Configure** on an environment card, then click **Edit Environment**, select your endpoint under **Trace Forwarding**, and click **Save Changes**.

## Add or edit an endpoint

Click **Add General Endpoint** or **Add GenAI Endpoint** to open the endpoint dialog. To edit an existing endpoint, click the pencil icon in its row. Both actions open the same dialog.

### Connection

| Field            | Required | Details                                                                                                               |
| ---------------- | -------- | --------------------------------------------------------------------------------------------------------------------- |
| **Name**         | Yes      | A human-readable label used to identify the endpoint in the list.                                                     |
| **Endpoint URL** | Yes      | The full URL of your OTLP receiver. Must be a valid HTTP or HTTPS URL.                                                |
| **Protocol**     | Yes      | **gRPC** (binary, efficient) or **HTTP** (REST-based). Defaults to gRPC for General Traces and HTTP for GenAI Traces. |

### Authentication

Select one of four options from the **Authorization Type** dropdown:

| Type               | Behavior                                                                                                                                |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------- |
| **None**           | No authentication headers are sent. Use only for internal or open endpoints.                                                            |
| **Bearer Token**   | Sends an `Authorization: <scheme> <token>` header. Requires a **Bearer Token**; **Bearer Scheme** is optional and defaults to `Bearer`. |
| **Basic Auth**     | Sends an `Authorization: Basic <base64>` header. Requires **Username** and **Password**.                                                |
| **Custom Headers** | Sends one or more arbitrary request headers. Add as many **Header Key** / **Header Value** pairs as needed.                             |

<Info>
  Secret values (tokens, passwords, header values) are encrypted at rest. When you reopen an endpoint for editing, existing secrets are masked as `****`. Click **Overwrite** to replace a secret, or leave it masked to keep the current value.
</Info>

### Optional fields

* **Additional headers** — Extra HTTP headers attached to every trace export request beyond those required for authentication. Click **+ Add Header** to add key-value pairs.
* **Resource attributes** — Key-value pairs added to the OTLP resource on every exported span. Use these to tag trace data with environment, team, or deployment metadata that your external platform can filter on (e.g. `environment = production`, `team = ai-platform`).

Click **Save & Test** to save the endpoint and verify connectivity immediately. General Trace endpoints also offer a **Save** option that skips the connection test. GenAI endpoints require **Save & Test** — connectivity must be verified before the endpoint is saved.

## Manage endpoints

### Test a connection

A connection test sends a sample OTLP trace to the configured endpoint. You can run a test two ways:

* Click **Save & Test** when creating or editing an endpoint.
* Click the **send icon** (Test Connection) in the endpoint list row.

The test times out after 30 seconds. A result modal shows latency, HTTP status code, trace ID, and span ID on success, or an error message and guidance on failure. From the modal you can **Edit Connection**, toggle **Enable** / **Disable**, or **Cancel** to close.

### Assign to an environment

After configuring and enabling an endpoint, assign it to one or more environments. Traces are only forwarded from environments where an endpoint is assigned.

1. Go to <img src="https://mintcdn.com/elementum/TFCVHNVI8zhq54sg/images/icons/settings.svg?fit=max&auto=format&n=TFCVHNVI8zhq54sg&q=85&s=3ffc43e8a0875412cc27335241aeb4c8" alt="Settings icon" className="inline-ui-icon" width="24" height="24" data-path="images/icons/settings.svg" /> **Organization Settings** > **Platform** > **Environments**.
2. Find the environment you want to forward traces from and click **Configure** on its card, then click **Edit Environment**.
3. In the **Trace Forwarding** section, select your configured endpoint from the list.
4. Click **Save Changes**.

Repeat for each environment you want to forward traces from.

### Delete an endpoint

1. Click the **trash icon** in the endpoint list row.
2. Confirm the deletion in the dialog that appears.

## Troubleshooting

<AccordionGroup>
  <Accordion title="Test returns 401 or 403">
    The credentials are invalid or expired. Re-enter the token or password and ensure the API key has the correct scopes.
  </Accordion>

  <Accordion title="Test returns 404">
    The endpoint URL is likely incorrect. Double-check the URL and, for HTTP endpoints, confirm the path includes `/v1/traces` or the equivalent path required by your platform.
  </Accordion>

  <Accordion title="Test times out after 30 seconds">
    A network or firewall issue is preventing the connection. Verify the endpoint host is reachable from the Elementum backend and check any applicable firewall rules.
  </Accordion>

  <Accordion title="Test passes but no data appears in the platform">
    Check that the endpoint is **Enabled** and assigned to the environment you are testing from. A passing test only confirms connectivity, not that traces are being produced — trigger an AI operation in Elementum to generate spans.
  </Accordion>

  <Accordion title="Trace Forwarding is not visible in the sidebar">
    The feature flag is not enabled for your organization. Contact your Elementum account team to enable the `AiTelemetryTableForwarding` feature flag.
  </Accordion>
</AccordionGroup>

## GenAI attribute reference

<Accordion title="View OpenTelemetry semantic conventions used in Elementum AI spans">
  ### LLM call spans

  These attributes appear on spans representing a single call to a language model. Span names follow the pattern `chat <model>`.

  | Attribute                        | Type          | Description                                                   |
  | -------------------------------- | ------------- | ------------------------------------------------------------- |
  | `gen_ai.system`                  | string        | The LLM provider (e.g. `aws.bedrock`, `openai`, `anthropic`). |
  | `gen_ai.request.model`           | string        | The model name that was requested.                            |
  | `gen_ai.response.model`          | string        | The model name that actually responded.                       |
  | `gen_ai.usage.input_tokens`      | integer       | Number of tokens in the prompt.                               |
  | `gen_ai.usage.output_tokens`     | integer       | Number of tokens in the completion.                           |
  | `gen_ai.prompt`                  | string (JSON) | Input messages as a JSON array of `{role, content}` objects.  |
  | `gen_ai.completion`              | string (JSON) | Output messages as a JSON array of `{role, content}` objects. |
  | `gen_ai.operation.name`          | string        | Always `"chat"` for LLM call spans.                           |
  | `gen_ai.response.finish_reasons` | string\[]     | Why the model stopped generating (e.g. `["stop"]`).           |
  | `gen_ai.response.id`             | string        | The provider's unique response identifier.                    |

  ### Tool execution spans

  These attributes appear on spans representing an agent executing a tool. Span names follow the pattern `execute_tool <tool_name>`.

  | Attribute               | Type   | Description                                               |
  | ----------------------- | ------ | --------------------------------------------------------- |
  | `gen_ai.tool.name`      | string | The function name of the tool that was called.            |
  | `gen_ai.tool.call.id`   | string | Links this execution back to the LLM's tool call request. |
  | `gen_ai.operation.name` | string | Always `"execute_tool"` for tool execution spans.         |

  ### Span tree structure

  Traces from Elementum AI operations are organized into a span hierarchy:

  | Interaction type                   | Span structure                                |
  | ---------------------------------- | --------------------------------------------- |
  | Single LLM call                    | `root → chat`                                 |
  | Tool-calling agent (one round)     | `root → chat → tool → chat`                   |
  | Multi-step agent (multiple rounds) | `root → chat → tool → chat → tool → chat → …` |
</Accordion>