Evaluation suites inside Elementum are a separate feature. See Test Agent Behavior. The traces on this page are for a platform you run yourself.
The platform-specific notes below come from each platform’s public documentation and from the attributes Elementum sends. Elementum does not certify any platform. Field mappings change between platform versions. Check the platform’s current OpenTelemetry documentation.
Before you start
- Read Prompt and response content. The platform stores prompts, replies, and tool results.
- Your platform must accept OTLP traces over the internet. Most accept OTLP over HTTP only. Check before you choose gRPC.
- Add a GenAI endpoint, not a General one, so the platform receives only AI spans. See Add a GenAI endpoint.
What an evaluation platform can read
Elementum sends the current OpenTelemetry names (
gen_ai.input.messages, gen_ai.output.messages, gen_ai.provider.name), not the older gen_ai.prompt, gen_ai.completion, and gen_ai.system. A platform that reads only the older names shows the spans without inputs and outputs. See Keys Elementum does not send.
W&B Weave
Settings for a Weave project on W&B’s multi-tenant cloud, from Weave’s documentation:
What to expect in Weave:
- Each
invoke_agentspan is a root, so each agent turn appears as its own trace, with itschatandexecute_toolcalls under it. - Weave reads
gen_ai.input.messagesandgen_ai.output.messagesas the inputs and outputs of eachchatcall. - Weave groups turns into threads by
gen_ai.conversation.id. - Token counts come from
gen_ai.usage.*.
Elementum has not verified this Weave mapping against a live trace. The inputs, outputs, threads, and token behavior above are taken from Weave’s documentation and source. Confirm them in your project with one test conversation before you build on them.
Build an evaluation in Weave
- Send a few real conversations through the agent, then open the project’s traces.
- Filter to the agent you want to evaluate, for example on
gen_ai.agent.name. - Select good and bad turns and add them to a dataset.
- Write a scorer and run an evaluation against the dataset. Weave’s evaluation guide covers scorers and comparisons.
Other platforms
Any platform that reads the OpenTelemetry GenAI conventions over OTLP can receive the same spans. Use the URL, path, authentication header, and project resource attributes that the platform documents for OTLP trace ingestion.
You can also send GenAI spans to a general tracing backend, such as Grafana Tempo or Datadog, through a General or GenAI endpoint. Using traces lists the field syntax for those backends.
Recipes
Token usage per agent. Sumgen_ai.usage.input_tokens and gen_ai.usage.output_tokens on chat spans, grouped by gen_ai.conversation.id, and join to invoke_agent for gen_ai.agent.name. Or group chat spans by gen_ai.request.model.
Slow model calls. Filter gen_ai.operation.name = chat and sort by duration. Compare gen_ai.request.model.
Tool failures. Filter gen_ai.operation.name = execute_tool with status ERROR, grouped by gen_ai.tool.name. The tool.execution.failed span event gives the kind of failure.
Replay a conversation. Filter on one gen_ai.conversation.id and order the chat spans by start time. The last gen_ai.input.messages holds the full history the model saw.
Regression check after an agent change. Build a dataset from traces taken before the change, then run the same inputs against the changed agent in your platform and compare scores.
Limits
- Delivery is best effort, as for all trace forwarding. A dataset built from traces can miss turns. See Delivery and failures.
- When the model calls several tools in one reply, one
execute_toolspan covers them all. Score tool use from thetool_callsingen_ai.output.messageswhen you need every call. - Streamed replies carry no
gen_ai.response.id. gen_ai.response.modelrepeats the requested model. It does not show a model substitution on the provider side.- Agent turns called over A2A are separate roots. They are not nested under the calling agent.
- Very long conversations can have truncated
gen_ai.input.messages.