Your Model Is the Coworker Who Overshares and Lies in the Same Breath
You know the one. Leaks a customer's phone number, then confidently invents a refund policy that does not exist, all in a single sentence, and never once notices. That is an unguarded LLM response talking to your users right now.
I built this discipline into regulatory-document pipelines for a living. The rule was simple. Nothing ships to a user or a compliance reviewer without knowing whether it is grounded in something real, and no chat transcript walks out the door carrying an SSN because nobody checked. guardrail-gate is that rule, rebuilt small enough to actually read.
It redacts the PII, checks whether every claim is backed by the sources you retrieved, throttles the clients trying to abuse you, and hands you one verdict: allowed or blocked, with the receipts attached. No mysterious "safety score" that means nothing.
Two Problems, Two Answers
Leaking private data and making things up are different failures. Blend them into one number and you lose the plot on both. So this checks them separately and tells you which one tripped.
• PII detection: pattern matching for emails, phone numbers, SSNs, credit cards, and IPs, and it hands back exactly which spans it scrubbed so you can audit the redaction instead of trusting it.
• Grounding: it breaks the answer into sentences and scores each one against your sources, then flags the response that snuck a hallucination in next to three true statements.
• Rate limiting: a per-client token bucket that returns a 429 before someone hammers you into a cloud bill.
• Hardening: optional API-key auth, size limits, a readiness probe, and error handling that never coughs up a stack trace to a stranger.
The Temptation I Said No To
Every instinct wanted to collapse PII and grounding into one clean confidence score. It demos beautifully. It is also useless, because when it fires you have no idea which failure you are looking at, and a guardrail you cannot explain just sells you false confidence. Two separate verdicts is uglier in the API and the only version that survives a real incident review.
I also wrote down what it misses, on purpose. The PII matcher will not catch a name buried in prose, that needs real NER. The grounding check will not catch a subtle fact bent inside an otherwise-solid sentence. A tool that pretends to catch everything is more dangerous than one that tells you its blind spots.
The Number
On the bundled benchmarks: 100% precision and recall on structured PII, and 83% accuracy telling a grounded answer apart from a hallucinated one.
Both Numbers Were Measured on the Easy Half
Those results were real, and I'd been careful about them. I'd even written the limitation into the README: lexical overlap "won't catch subtle factual drift." Right instinct. Wildly too gentle. Here is what was actually getting through:
Source: "Refunds are processed within 10 business days of the return being received."
Response: "Refunds are not processed within 10 business days of the return being received."
Overlap: 0.90. Verdict: grounded.
One inserted word inverts a refund policy and moves the score by a rounding error. Six of eight hallucinations of that shape passed, including "$49 per month" becoming "$49 per year" at 0.89 overlap, and "3 to 5 business days" becoming "30 to 50."
And this isn't an edge case, it's the central one. A model generating from retrieved context doesn't invent new vocabulary. It recombines the vocabulary sitting in front of it. Which means the hallucinations that actually happen in production are precisely the ones bag-of-words similarity scores highest. My benchmark missed it because I'd written the hallucinated cases with fresh words, "free for the first year," "same-day refunds," which overlap already handles fine. I'd tested the failure mode that doesn't occur.
The fix isn't a better similarity metric, it's checking what a claim actually asserts: polarity, quantities with their units, and the prepositional conditions that say when a claim applies, with overlap kept only as the floor. "Within 10 days of the return being received" and "of the order being placed" start the clock at completely different events, and that's now a blocking difference. Genuine paraphrases still pass, which is the constraint that makes it usable at all: "once your order is confirmed" matches "after order confirmation" because conditions compare as stemmed word sets rather than strings.
The Redactor Was Wrong in Both Directions at Once
The PII side scored 100/100 on textbook formatting and 29% recall on how people actually type. It missed "jane dot doe at example dot com," missed a spaced-out SSN, missed every international phone number. Meanwhile it redacted "Version 1.2.3.4" as an IP address and order number "4111 1111 1111 1112" as a credit card. That second failure is the one that actually kills a guardrail: mangle enough order numbers and someone turns the redactor off, and then it protects nothing.
The rewrite validates instead of pattern-matching. A credit card isn't sixteen digits, it's sixteen digits that pass the Luhn checksum and begin with a real issuer prefix. An SSN isn't three-two-four digits, because the SSA never issues 000, 666, or 900-999 as an area number. A phone number obeys the NANP rule that area code and exchange both start 2-9, which is exactly what makes "100-200-3000 units" a measurement. Those rules cost nothing and are what make the extra recall affordable.
Held Out, Run Once
I built both replacements against my own adversarial corpus, so scoring 100% on it proves nothing. So I wrote a separate held-out set afterwards with the code frozen and ran it exactly once: PII went from 55%/75% precision and recall to 100%/100%. Grounding went from 46% to 85%. Those two numbers are the only ones here I'd defend, because they're the only ones measured on data the code never saw.
v2 misses two held-out cases, both the same shape: "on weekdays" rewritten as "on weekends." A scope substitution with no number, no negation, and no condition to catch it. I've left it unpatched on purpose. Fixing it after reading the holdout is how a holdout quietly stops being one, and it stands in the README as the honest description of what this checks and what it doesn't.
Three bugs surfaced while building it, all in my own new code: the quantity extractor swallowed the word "per," so "$49 per year" and "$49 per month" both collapsed to a bare 49 and compared equal; "monthly" wasn't recognized as a unit at all; and the stemmer reduced "received" to "receiv" while leaving "receive" untouched, which meant the exact paraphrase it existed to accept was still being rejected. 76 tests now, and CI fails the build if a hallucination starts getting through or the redactor starts eating order numbers again.