The Bouncer Between Your Model and Your User
Picture the coworker who overshares personal details and makes things up, in the same sentence, without noticing either one. That's an unguarded LLM response. I built the same discipline into regulatory-document LLM pipelines at work: you don't ship a response to a user, or a compliance reviewer, without knowing whether it's grounded in something real, and you don't let a chat transcript leak an SSN because nobody thought to check. guardrail-gate is that discipline rebuilt from scratch as something small, runnable, and inspectable.
It redacts PII, checks whether each claim in a response is actually supported by the sources you retrieved, and rate-limits abusive clients, then hands back a single allowed/blocked decision with the reasons attached instead of a vague combined "safety score."
Technical Architecture
PII and hallucination are different failure modes, and conflating them produces worse guardrails, not better ones, so the service checks them separately and reports both.
• PII detection: regex-complete structured pattern matching for emails, phone numbers, SSNs, credit cards, and IPs, with redacted spans returned so the caller can audit exactly what was removed.
• Grounding check: splits the response into sentences and scores each against the retrieved source set, flagging low-grounding responses instead of letting a hallucinated claim through with a true one.
• Rate limiting: a per-client token bucket returns 429 before an abusive client can hammer the service.
• Production hardening: optional API-key auth, request size limits, readiness probe, and structured error handling that never leaks a stack trace.
Challenges & Solutions
The real challenge was resisting the urge to combine PII and grounding into one confidence score. It reads cleaner in a demo, but in practice it hides which failure mode actually triggered, and a guardrail you can't explain is worse than no guardrail because it creates false confidence. Keeping them as two separately reported checks meant more surface area in the API response, but it's the only version that's actually debuggable in production.
I was also upfront in the README about what the benchmark doesn't catch: the PII regex won't catch a name mentioned in prose (that needs real NER), and the lexical grounding check won't catch subtle factual drift within an otherwise-grounded sentence. Disclosing the limits was more useful than a benchmark number that implies the tool catches everything.
Impact & Results
On the labeled benchmarks bundled with the repo: 100% precision and recall on structured PII (emails, phone numbers, SSNs, credit cards, IPs), and 83% accuracy separating grounded responses from hallucinated ones.
20 tests pass in CI across the full Python matrix, plus a live Docker smoke test that boots the container and hits the real endpoint with a PII-and-grounding example end to end.