> ## Documentation Index
> Fetch the complete documentation index at: https://docs.elementum.io/llms.txt
> Use this file to discover all available pages before exploring further.

# GenAI Tracing

> What General and GenAI endpoints receive for AI agents, including model calls, tools, voice, and prompt content.

Elementum can send traces of its AI agents to your OpenTelemetry backend or to an AI evaluation platform, such as W\&B Weave. Each agent turn arrives as a small tree of spans: the agent invocation, every call to the language model, and every tool the agent ran. The spans follow the OpenTelemetry GenAI semantic conventions, so the `gen_ai.*` attributes carry the model, token usage, the messages sent to the model, and its reply.

This page describes the spans and what they contain. For every attribute, see the [GenAI attribute reference](https://docs.elementum.io/administration/trace-forwarding/genai-attribute-reference.md). To use the traces for evaluation, see [Evaluating AI with traces](https://docs.elementum.io/administration/trace-forwarding/evaluating-ai.md).

> [!WARNING]
>
> LLM call and tool spans carry the full text of prompts, replies, tool arguments, and tool results. Both GenAI and General endpoints receive them. Read [Prompt and response content](#prompt-and-response-content) before you connect an endpoint.

## Which endpoint receives what

| Spans | General endpoint | GenAI endpoint |
| - | - | - |
| Automation spans (trigger, run, actions, HTTP calls) | Yes | No |
| AI agent spans: agent invocation, LLM calls, tool runs | Yes | Yes |
| Voice call spans listed under [Voice calls](#voice-calls) | Yes | Yes |
| Other AI activity: conversation and message handling, agent and tool changes, email and voice call steps | Yes | No |
| Elementum internal platform spans | No | No |

A GenAI endpoint receives only the spans that describe AI interactions. A General endpoint receives everything a GenAI endpoint does, plus automation spans and other AI activity spans. Use a GenAI endpoint when the receiver is an evaluation platform that should see only model and agent activity.

Both endpoint types are set up the same way. See [Add a GenAI endpoint](https://docs.elementum.io/administration/trace-forwarding/set-up.md#add-a-genai-endpoint).

## Span types

| Span name | One per | `gen_ai.operation.name` | `openinference.span.kind` |
| - | - | - | - |
| `invoke_agent <agent name>` | Agent turn: a message sent to an agent, a streamed reply, or a tool confirmation | `invoke_agent` | `AGENT` |
| `chat <model>` | Call to the language model | `chat` | `LLM` |
| `execute_tool <tool name>` | Set of tool calls the model asked for in one reply | `execute_tool` | `TOOL` |
| `a2a.message.handle` | Message received from another agent over the Agent2Agent (A2A) protocol | none | none |

Span names include the agent name, model, or tool name when Elementum knows it. Otherwise the name falls back to a fixed name:

| Usual name | Fallback name |
| - | - |
| `invoke_agent <agent name>` | `invoke_agent` |
| `chat <model>` | `chat.llm.call` for a single reply, or `chat.llm.stream` for a streamed reply |
| `execute_tool <tool name>` | `tool.dispatch` |

To group spans, filter on `gen_ai.operation.name` or `openinference.span.kind`, not on the span name. Span names change with the model and the tool.

All of these spans have kind `INTERNAL`, resource `service.name = elementum`, and instrumentation scope `elementum`, like automation spans.

## Span tree

One agent turn in which the model called one tool:

```text theme={null}
invoke_agent Support Assistant        AGENT   gen_ai.conversation.id=c0000000-...
├── chat gpt-4o                       LLM     model replies with a tool call
├── execute_tool searchRecords        TOOL    gen_ai.tool.call.id=call_1
└── chat gpt-4o                       LLM     model writes the final answer
```

* Every LLM call and tool run of the turn is a direct child of `invoke_agent`. They are siblings, in start-time order.
* A turn with no tool calls has one `chat` span. A turn with several tool rounds alternates `chat` and `execute_tool`.
* When the model asks for several tools in one reply, one `execute_tool` span covers all of them. Its `gen_ai.tool.*` attributes describe only one of those tools. The full list is in the `tool_calls` of the previous `chat` span's `gen_ai.output.messages`.

### Parents on each endpoint type

Elementum sends each endpoint a tree made only of the spans that endpoint receives. A span's parent is its nearest ancestor that the endpoint also receives.

| Endpoint | Parent of `invoke_agent` | Parent of `chat` and `execute_tool` |
| - | - | - |
| GenAI | None. Every agent turn is a root span. | `invoke_agent` |
| General | The surrounding Elementum request span, for example the span for the message being created | `invoke_agent` |

Agent turns of one conversation usually arrive as separate traces. They share `gen_ai.conversation.id`. Use that attribute to see a whole conversation.

When one agent calls another over A2A, the called agent's `invoke_agent` is a separate root on a GenAI endpoint. It is not nested under the calling agent. Use `gen_ai.conversation.id` and timing to connect them.

## Voice calls

Voice agents produce spans too. Two kinds of call exist, depending on how the voice agent is set up.

### Calls through Elementum's voice provider

| Span name | What it is |
| - | - |
| `invoke_agent <model>` | The end-of-call summary. Carries the call transcript, the agent's last reply, token usage, and the provider's call summary. Named `voice.call.summary` if no model or agent name is known. |
| `voice.tool.call` | A tool call the voice agent made. The `execute_tool` span of the tool run is its child. |
| `voice.authentication` | Caller authentication. |

### Realtime voice calls

Each realtime call starts a new trace:

```text theme={null}
voice.realtime.call                   one per call
└── voice.realtime.session            invoke_agent: the agent conversation
    ├── voice.realtime.caller         one per caller turn
    ├── voice.realtime.response       chat: one per spoken model reply
    └── voice.realtime.dictated       a scripted line, such as a greeting
```

Realtime spans carry timings, turn outcomes, and model details, but no transcript text. Voice spans also carry `voice.*` attributes. See [Voice attributes](https://docs.elementum.io/administration/trace-forwarding/genai-attribute-reference.md#voice-attributes).

## Span status and errors

| Status | When |
| - | - |
| `OK` | The step completed. `elementum.action.status` is `SUCCESS`. |
| `ERROR` | The step failed. `elementum.action.status` is `FAILURE`, when set. |
| `UNSET` | No status was recorded. |

On a failed span, `elementum.error.type` is the class name of the error and `elementum.error.message` is the error's message text. This differs from automation spans, which carry only a fixed label. An AI error message can quote details from the model provider's response.

A failed tool run has status `ERROR` and a span event, `tool.execution.failed`, with the tool name, the kind of failure, and its duration.

### Span events

Unlike automation spans, AI spans can carry span events. They mark progress steps of the turn, for example `message.llm.call.completed`. Their names and attributes are not a stable contract and can change. Events that Elementum marks as sensitive are never sent. Exception events and stack traces are never sent.

## Prompt and response content

GenAI spans exist so you can evaluate what your agents said. They carry content.

| Content | Span | Attribute |
| - | - | - |
| Every message sent to the model: the agent's instructions (system message), the conversation history, the user's message, and earlier tool results | `chat` | `gen_ai.input.messages` |
| The model's reply, including any tool calls and their arguments | `chat` | `gen_ai.output.messages` |
| Arguments passed to a tool, and the tool's result | `execute_tool` | `gen_ai.tool.call.arguments`, `gen_ai.tool.call.result`, `input.value`, `output.value` |
| The agent's description | `invoke_agent` | `gen_ai.agent.description` |
| Voice call transcript, last agent reply, and call summary | Voice end-of-call summary | `gen_ai.input.messages`, `gen_ai.output.messages`, `voice.call.analysis.summary` |

How content is handled:

* **Content is always included.** There is no setting to send these spans without it. To keep content out of a backend, drop or scrub the content attributes in an OpenTelemetry Collector in front of that backend, or do not forward to it.
* **General and GenAI endpoints receive the same content.** A General endpoint that receives AI spans stores prompts and replies too.
* **Elementum does not redact personal data** from prompts, replies, or tool results. Tool results can contain record data that the tool read.
* **Voice PINs are masked.** In voice transcripts and summaries, PINs a caller gives are replaced by a mask. Each transcript message is cut to 1,000 characters, and tool arguments are cut to 2,000 characters.
* **Very long values can be cut off.** Elementum limits the size of one attribute value. A very long conversation history can arrive truncated.

### What is never sent

* Elementum internal platform spans: database calls, queues, and internal service calls.
* Organization and environment ids. Spans are routed by them, but they are not attributes.
* Inputs and outputs Elementum records for its own use on other AI activity spans.
* Exception events and stack traces.
* Attributes that are not in the [GenAI attribute reference](https://docs.elementum.io/administration/trace-forwarding/genai-attribute-reference.md). Elementum sends only attribute keys under `gen_ai.`, `elementum.`, `voice.`, `llm.`, `openinference.`, `input.`, and `output.`.

## Personal data

| Data | Where |
| - | - |
| What users, callers, and agents said | `gen_ai.input.messages`, `gen_ai.output.messages` |
| Record data read or written by tools | `gen_ai.tool.call.arguments`, `gen_ai.tool.call.result`, `input.value`, `output.value` |
| Agent names, descriptions, and instructions written in Elementum | `gen_ai.agent.name`, `gen_ai.agent.description`, and system messages in `gen_ai.input.messages` |
| Error text | `elementum.error.message` |
| Ids | `gen_ai.agent.id`, `gen_ai.conversation.id`, `voice.call.id` |

Treat any endpoint that receives AI spans as a store of conversation data. Check its access controls, region, and retention before you connect it, and apply the same rules you apply to conversation history.

Next: [GenAI attribute reference](https://docs.elementum.io/administration/trace-forwarding/genai-attribute-reference.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.