A buyer-oriented decision framework. Everything below is backed by the public N=500 benchmarks. If any claim doesn't match your data, the harness is Apache 2.0 — reproduce or disprove it on your corpus. The engineering-grade source lives in USECASES.md.
All numbers from the public benchmarks runs at N=500 (arXiv at N=100 — full available test slice). Qwen2.5-VL-7B throughout.
Across all nine public benchmarks, Bernoulli's NLL is 7–19× better than the same backbone used as a text-and-parse classifier. If your code is if confidence > 0.9: auto_approve, a miscalibrated 0.9 is a product bug — your false-positive rate quietly climbs.
arXiv papers published 2026-09-18 to 2026-10-06 — strictly after both Qwen's and DeBERTa's training cutoffs. Bernoulli 0.95 vs DeBERTa-zeroshot 0.53 — a 42-point gap on content neither model could have memorized. Our architecture generalizes; a zero-shot NLI encoder tuned on fixed data doesn't.
Zero-shot NLI encoders like DeBERTa only score natural-language label hypotheses — they don't handle binary Yes/No. On PAWS (adversarial paraphrase binary): Bernoulli 0.846 / NLL 0.376; BGE-m3+LR crashes to 0.552; DeBERTa n/a by design. For guardrails, policy checks, is-this-X questions — zero-shot encoders are off the map and prompt-and-parse is poorly calibrated.
Triage, LLM-as-judge, moderation, code review — real workloads need several classifications per input. Prompt-and-parse makes one HTTP call per question; Bernoulli processes the state once and shares it across questions via prefix caching. On A10G 24GB: 1 question in 76 ms, 5 questions in 368 ms, 20 questions in 1466 ms — roughly ~70 ms per additional question.
Apache 2.0 code, open-weight model, backbone-agnostic (any VL model with a BERNOULLI_MODEL_ID swap). Three deployment modes: self-host in your VPC (flagship), managed-by-us-on-your-cloud (BYOC), or hosted-by-us. The self-host path is zero outbound calls at inference time. If data residency is a hard requirement, we're in a different category from any hosted API.
If you recognize your workload in any of these, pick the named alternative — Bernoulli isn't the right tool for the job.
Train a classifier. On tasks with labeled data: Banking77 (77-way) BGE+LR 0.95 vs us 0.57, CLINC150-OOS (151-way) BGE+LR 0.80 vs us 0.45, ToxicChat BGE+LR 0.78 vs us 0.69. A trained classifier has fundamentally better signal than any zero-shot approach.
DeBERTa-v3-zeroshot was NLI-tuned on datasets covering a lot of academic benchmark content. SST-2 +1.6 pt, AG News +4.6 pt, Banking77 accuracy +6.8 pt. The arXiv post-cutoff crash (0.53) tells you this is training-overlap, not generalization — but if your task is similar, it works.
For "flag unsafe prompts" at scale where unsafe-recall is the top metric, specialist safety models are tuned specifically for this. Our XSTest numbers tell the honest story: <1% false-refusal (safe-by-default) but unsafe-recall only 0.28 — we're a first-pass gate, not a final answer.
CLINC150-OOS tests OOS detection in a 150-intent space. Our oos_auroc = 0.48 — essentially coin flip. BGE-m3+LR hits 0.78 (not by predicting OOS, but by ranking in-scope confidences well enough that a threshold works). Real limitation of our prompt framing today.
Our chunked scoring (for choice tasks with >26 options) splits labels into chunks of ≤26 and softmaxes globally. Banking77 reorder stability: 0.07 (DeBERTa and BGE+LR hit 1.00). Architectural, not a bug — cross-chunk debias isn't shipped yet.
Summaries, explanations, rewritten emails, code suggestions. Bernoulli is for decisions, not content. Use a generative LLM. If you want both a decision and an explanation, run them as two separate calls — one Bernoulli, one chat — and keep the clean separation.
Bernoulli assumes your decision has a fixed enumeration of possible answers. If the right answer is "any of these 10,000 product SKUs" or "generate a new SKU name," that's a different tool.
If one of these fits better than Bernoulli, use it. We'd rather you solve your problem than use our thing.
NLI-tuned encoder. Treats classification as "entailment of a hypothesis." Choice tasks only; binary Yes/No is unsupported by design.
Sentence embedder + classical ML on top. Needs 500–2000+ labeled examples per task. Wins decisively on long-tail labeled tasks.
Ask an LLM in natural language, parse the completion. Current default for "add intelligence to this backend." Terrible calibration (7–19× worse NLL than us).
Fine-tune a classifier on your labeled data. Highest accuracy ceiling, biggest ops cost. The "if money and data are no object" answer.
Open-weight models fine-tuned specifically for safety classification. Report token probabilities over safe/unsafe labels — apples-to-apples with Bernoulli when safety is the only question.
Still legitimate in 2026. If your decision reduces to counting words from a known lexicon or matching regex patterns, that's often faster, cheaper, and more auditable than any ML approach.
Equal time to the downsides. Everything here is published in the per-benchmark result files and in the benchmarks README caveats.
All public numbers are from Qwen2.5-VL-7B. A one-shot H100 rental of the 32B variant lifted the hardest JevBench slice by ~18 points — see the H100 writeup. Expect every row here to improve on the production tier.
Every result row carries a reproducibility footer (model revision, dataset hash, seed, hardware) — any number is reproducible or disprovable. The JevBench submission is in the public leaderboard's queue for an external re-run; this page updates once that posts.
WildGuardTest is HF-gated; a Llama Guard 3 vs Bernoulli comparison hasn't shipped. If your workload is production safety where unsafe-recall dominates, we're honestly not pretending to compete with the specialists.
oos_auroc = 0.48 at N=500. On long-tail intent sets with an OOS category, our prompt framing doesn't carry enough signal. Investigation-pending, not shipped.
Three ways forward: try the live demo, run the benchmarks on your own data, or book a 20-minute fit review with us.