The Final Answer Hides The Path
An agent can return the right answer after a bad tool loop, a slower model call, or twice the token spend. Answer-only tests miss the behavior that becomes your production bill and latency budget.
spanjudge accepts OpenTelemetry Protocol JSON, stores agent and tool spans in SQLite, and rebuilds each trace into latency, cost, error, token, and evaluation metrics.
Make The Trace A Release Contract
A JSON policy sets ceilings for p95 latency, error rate, and average cost plus a floor for evaluation score. The CLI exits non-zero when any limit fails, so the same check works locally, in Docker, and in GitHub Actions.
The repository ships the collector, HTTP API, responsive dashboard, CLI, SQLite store, Docker image, test suite, and CI matrix. No hosted observability service is required to run the complete demo.
The Demo
Six spans reconstruct three agent traces. The fixture reports 980 ms p95 latency, 0% trace error rate, $0.0039 average cost, 0.88 average evaluation score, and 1,513 total tokens. All four release checks pass.
This is a focused OTLP/HTTP JSON receiver, not a replacement for the full OpenTelemetry Collector. Production ingestion still needs authentication, redaction, retention, and transport security.
A Real Bug: Two Traces, One Row
The SQLite table storing spans used span_id alone as its primary key. The OTLP spec only guarantees a span ID is unique within its own trace, not globally — so two different traces that happened to reuse the same span ID would silently overwrite each other on ingest. One trace's spans, cost, latency, and evaluation data would vanish from the store with no error anywhere in the path. For a release gate, that means a policy check could pass against silently incomplete data.
Verified directly: ingesting two spans with the same span ID but different trace IDs left only one trace queryable afterward. Fixed by keying the table on (trace_id, span_id) instead, matching what the spec actually promises, plus a startup migration that safely upgrades any existing database still on the old schema without losing data. Covered by a dedicated test file: the original collision, the fix, that legitimate same-trace span re-ingestion (a client retry) still works, and that the legacy-schema migration is correct and idempotent.
Standards Basis
Built around the OpenTelemetry GenAI attribute registry for agent, tool, token, and evaluation telemetry.