The Final Answer Hides The Path
An agent can return the right answer after a bad tool loop, a slower model call, or twice the token spend. Answer-only tests miss the behavior that becomes your production bill and latency budget.
spanjudge accepts OpenTelemetry Protocol JSON, stores agent and tool spans in SQLite, and rebuilds each trace into latency, cost, error, token, and evaluation metrics.
Make The Trace A Release Contract
A JSON policy sets ceilings for p95 latency, error rate, and average cost plus a floor for evaluation score. The CLI exits non-zero when any limit fails, so the same check works locally, in Docker, and in GitHub Actions.
The repository ships the collector, HTTP API, responsive dashboard, CLI, SQLite store, Docker image, test suite, and CI matrix. No hosted observability service is required to run the complete demo.
The Demo
Six spans reconstruct three agent traces. The fixture reports 980 ms p95 latency, 0% trace error rate, $0.0039 average cost, 0.88 average evaluation score, and 1,513 total tokens. All four release checks pass.
This is a focused OTLP/HTTP JSON receiver, not a replacement for the full OpenTelemetry Collector. Production ingestion still needs authentication, redaction, retention, and transport security.
Standards Basis
Built around the OpenTelemetry GenAI attribute registry for agent, tool, token, and evaluation telemetry.