Problem
Any user-facing LLM product gets probed: jailbreaks, prompt injection, people trying to turn your support bot into a free frontier model. The quieter risk is compliance - a bank's assistant being steered into citing data it was never supposed to surface, or into statements that make the brand look bad in a screenshot. Screening every message with a frontier model roughly doubles your cost and latency - and trusting the system prompt to hold is not a strategy.
What we built
Policy Guard: a fine-tuned Qwen classifier that runs in front of the frontier call. Policies are prompts, not hardcoded classes - you write what a violation means for your product (prompt injection, data exfiltration, off-brand statements, whatever compliance requires) and the guard enforces it; multiple policies evaluate in parallel on every message, Promise.all style. Self-hosted, so suspicious traffic gets analysed without ever leaving the infrastructure.
Findings
Adversarial hardening
The naive fine-tune looked perfect in-distribution - and caught only 68.8% of real-world jailbreaks. Two rounds of hardening (feeding it its own misses, then obfuscated benign controls) closed that to 170/170.
Held-out performance
99.6% catch rate at a 0.25% false-positive rate on a 1,320-example held-out test.
Small beats big
At matched precision, the 8×-smaller fine-tune catches 96.2% of compliance breaches versus the 35B incumbent's 84.6%. The whole adapter is 146 MB of LoRA riding on a 4B base.
Thresholds are product decisions
Five tuned operating thresholds ship as selectable modes - from a greedy default to a strict-exfiltration mode with zero measured false positives. Latency and per-check cost are being re-benchmarked and will follow.
What we guarded against
| Attack vector | What it looks like | Result |
|---|---|---|
| Prompt injection | Instructions smuggled into user input to override the system | 99.6% catch @ 0.25% FP (n=1,320 held-out) |
| Real-world jailbreaks | Published working attacks from the L1B3RT4S corpus | 170/170 after two hardening rounds |
| Obfuscated variants | The same attacks disguised - encodings, paraphrase, fragmentation | Recall 1.0 at 0.57% FP (530 variants) |
| Data exfiltration | Coaxing the model into citing internal data it should never surface | Perfect at the strict threshold, zero measured FP |
| Compliance breaches | Off-policy statements - the screenshot that makes a bank look bad | 96.2% recall at matched precision (n=178) |
Hardening: before and after
8× smaller, better recall
99.6%
held-out catch rate (1,320 examples)
0.25%
false positives at production threshold
170/170
real-world jailbreaks caught after hardening
Use cases
Anything that puts an LLM in front of strangers: customer-facing chatbots, in-product assistants, public API endpoints wrapping a frontier model. It earns its keep internally too - screening prompts before they reach tools that touch tenant data, and routing obvious abuse away from expensive models entirely. The same recipe - a small classifier, adversarially hardened on its own misses - transfers to content moderation and fraud-signal screening.