> ## Documentation Index
> Fetch the complete documentation index at: https://docs.elementum.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluating AI with Traces

> Connect an evaluation platform such as W&B Weave and score agent behavior from the spans Elementum sends.

GenAI traces give an evaluation platform the conversations to score: what each agent was asked, what the model answered, which tools it called, and how many tokens it used. This page explains how to connect a platform, what it can read from Elementum spans, and how to build evaluations from them.

> [!NOTE]
>
> Evaluation suites inside Elementum are a separate feature. See [Test Agent Behavior](https://docs.elementum.io/ai-agents/agents-evaluations.md). The traces on this page are for a platform you run yourself.

> [!NOTE]
>
> The platform-specific notes below come from each platform's public documentation and from the attributes Elementum sends. Elementum does not certify any platform. Field mappings change between platform versions. Check the platform's current OpenTelemetry documentation.

## Before you start

* Read [Prompt and response content](https://docs.elementum.io/administration/trace-forwarding/genai-tracing.md#prompt-and-response-content). The platform stores prompts, replies, and tool results.
* Your platform must accept OTLP traces over the internet. Most accept OTLP over HTTP only. Check before you choose **gRPC**.
* Add a **GenAI** endpoint, not a General one, so the platform receives only AI spans. See [Add a GenAI endpoint](https://docs.elementum.io/administration/trace-forwarding/set-up.md#add-a-genai-endpoint).

## What an evaluation platform can read

| Evaluation need | Elementum attribute |
| - | - |
| Input to score | `gen_ai.input.messages` on `chat` spans |
| Output to score | `gen_ai.output.messages` on `chat` spans |
| Tool use: which tool, with what arguments, and what came back | `gen_ai.tool.name`, `gen_ai.tool.call.arguments`, `gen_ai.tool.call.result` on `execute_tool` spans |
| Group turns into a conversation | `gen_ai.conversation.id` |
| Compare agents | `gen_ai.agent.id`, `gen_ai.agent.name` |
| Compare models | `gen_ai.request.model`, `gen_ai.provider.name` |
| Cost and size | `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.usage.cache_read.input_tokens` |
| Latency | Span duration of `chat` and `invoke_agent` |
| Failures | Status `ERROR`, `elementum.error.type`, `elementum.error.message` |
| Classify spans | `gen_ai.operation.name`, `openinference.span.kind` |

Elementum sends the current OpenTelemetry names (`gen_ai.input.messages`, `gen_ai.output.messages`, `gen_ai.provider.name`), not the older `gen_ai.prompt`, `gen_ai.completion`, and `gen_ai.system`. A platform that reads only the older names shows the spans without inputs and outputs. See [Keys Elementum does not send](https://docs.elementum.io/administration/trace-forwarding/genai-attribute-reference.md#keys-elementum-does-not-send).

## W\&B Weave

Settings for a Weave project on W\&B's multi-tenant cloud, from Weave's documentation:

| Elementum field | Value |
| - | - |
| **Endpoint URL** | `https://trace.wandb.ai/otel/v1/traces`. Dedicated and self-managed W\&B use `https://<your-subdomain>.wandb.io/traces/otel/v1/traces`. |
| **Protocol** | **HTTP**. Weave does not accept gRPC. |
| **Authorization Type** | **Custom Headers**, with **Header Key** `wandb-api-key` and your W\&B API key as **Header Value**. |
| **Resource Attributes** | `wandb.entity` set to your W\&B team or user name, and `wandb.project` set to the project that should store the traces. |

What to expect in Weave:

* Each `invoke_agent` span is a root, so each agent turn appears as its own trace, with its `chat` and `execute_tool` calls under it.
* Weave reads `gen_ai.input.messages` and `gen_ai.output.messages` as the inputs and outputs of each `chat` call.
* Weave groups turns into threads by `gen_ai.conversation.id`.
* Token counts come from `gen_ai.usage.*`.

> [!NOTE]
>
> Elementum has not verified this Weave mapping against a live trace. The inputs, outputs, threads, and token behavior above are taken from Weave's documentation and source. Confirm them in your project with one test conversation before you build on them.

### Build an evaluation in Weave

1. Send a few real conversations through the agent, then open the project's traces.
2. Filter to the agent you want to evaluate, for example on `gen_ai.agent.name`.
3. Select good and bad turns and add them to a dataset.
4. Write a scorer and run an evaluation against the dataset. Weave's evaluation guide covers scorers and comparisons.

## Other platforms

Any platform that reads the OpenTelemetry GenAI conventions over OTLP can receive the same spans. Use the URL, path, authentication header, and project resource attributes that the platform documents for OTLP trace ingestion.

| Platform reads | Works with Elementum spans |
| - | - |
| OpenTelemetry GenAI conventions, current names | Yes. Inputs, outputs, model, usage, agent, tool, and conversation are available. |
| OpenInference conventions | Partly. `openinference.span.kind`, `llm.provider`, `input.value`, and `output.value` are set on tool spans. LLM call content is only under `gen_ai.*`. |
| Only the older `gen_ai.prompt` and `gen_ai.completion` | Inputs and outputs are missing. Map the current names in an OpenTelemetry Collector, or ask the vendor to read the current names. |

You can also send GenAI spans to a general tracing backend, such as Grafana Tempo or Datadog, through a General or GenAI endpoint. [Using traces](https://docs.elementum.io/administration/trace-forwarding/using-traces.md) lists the field syntax for those backends.

## Recipes

**Token usage per agent.** Sum `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens` on `chat` spans, grouped by `gen_ai.conversation.id`, and join to `invoke_agent` for `gen_ai.agent.name`. Or group `chat` spans by `gen_ai.request.model`.

**Slow model calls.** Filter `gen_ai.operation.name = chat` and sort by duration. Compare `gen_ai.request.model`.

**Tool failures.** Filter `gen_ai.operation.name = execute_tool` with status `ERROR`, grouped by `gen_ai.tool.name`. The `tool.execution.failed` span event gives the kind of failure.

**Replay a conversation.** Filter on one `gen_ai.conversation.id` and order the `chat` spans by start time. The last `gen_ai.input.messages` holds the full history the model saw.

**Regression check after an agent change.** Build a dataset from traces taken before the change, then run the same inputs against the changed agent in your platform and compare scores.

## Limits

* Delivery is best effort, as for all trace forwarding. A dataset built from traces can miss turns. See [Delivery and failures](https://docs.elementum.io/administration/trace-forwarding/set-up.md#delivery-and-failures).
* When the model calls several tools in one reply, one `execute_tool` span covers them all. Score tool use from the `tool_calls` in `gen_ai.output.messages` when you need every call.
* Streamed replies carry no `gen_ai.response.id`.
* `gen_ai.response.model` repeats the requested model. It does not show a model substitution on the provider side.
* Agent turns called over A2A are separate roots. They are not nested under the calling agent.
* Very long conversations can have truncated `gen_ai.input.messages`.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.