The Final Answer Hides the Path
An agent can return the right answer after a bad tool loop, a slower model call, or twice the token spend. Answer-only tests miss exactly the behavior that turns into your production bill and your latency budget.
spanjudge accepts OpenTelemetry Protocol JSON, stores agent and tool spans in SQLite, and rebuilds each trace into latency, cost, error, token, and evaluation metrics.
Make the Trace a Release Contract
A JSON policy sets ceilings for p95 latency, error rate, and average cost, plus a floor for evaluation score. The CLI exits non-zero when any limit fails, so the same check runs locally, in Docker, and in GitHub Actions.
It ships the collector, an HTTP API, a dashboard, the CLI, the SQLite store, a Docker image, and tests. No hosted observability service is needed to run the whole thing.
The Demo
Six spans rebuild three healthy agent traces: 980 ms p95 latency, 0% errors, $0.0039 average cost, 0.88 average evaluation score. All four release checks pass.
The second fixture is the one the project exists for. Same three traces and the same 0.88 evaluation score, so every answer still passes. But the agent now repeats its tool call and every span takes twice as long. p95 latency hits 1,960 ms and average cost $0.0082. The gate fails both and exits 1. An answer-only test ships that release. This one blocks it, and CI fails the build if it ever stops blocking it.
It's a focused OTLP/HTTP JSON receiver, not a replacement for the full OpenTelemetry Collector. Production ingestion still needs authentication, redaction, retention, and transport security.
Three Bugs, Two of Them Silent
The span table used span_id alone as its primary key, but OTLP only guarantees a span ID is unique within its own trace. Two traces that reused a span ID silently overwrote each other on ingest. One trace's spans, cost, and evaluation data vanished with no error anywhere, so a release gate could pass against incomplete data. It's now keyed on (trace_id, span_id), as the spec promises, with a startup migration for existing databases. Tests cover the collision, a legitimate client retry, and an idempotent migration.
The second turned up while auditing this page: the Docker image crashed on a plain docker run with unable to open database file, because /data only existed when Compose mounted a volume. The store now creates its directory and the image declares the volume. Fixing that exposed a third: the dashboard lived in a web/ folder outside the Python package, so the image served a FileNotFoundError at /. It ships inside the package now, and CI builds the image and fetches both the API and the page before it calls the build green. 10 tests pass across Python 3.10 through 3.13.
Standards Basis
Built around the OpenTelemetry GenAI attribute registry for agent, tool, token, and evaluation telemetry.
Try It
git clone https://github.com/ahmeddoghri/spanjudge && cd spanjudge
pip install -e .
spanjudge --database demo.db ingest demo/traces.json
spanjudge --database demo.db evaluate demo/policy.json # passes
spanjudge --database bad.db ingest demo/regressed_traces.json
spanjudge --database bad.db evaluate demo/policy.json; echo $? # 1
spanjudge --database demo.db serve # dashboard on :4318
Or docker compose up --build, and point any OTLP/HTTP JSON exporter at /v1/traces.