When to use

Is Bernoulli right for your workload?

A buyer-oriented decision framework. Everything below is backed by the public N=500 benchmarks. If any claim doesn't match your data, the harness is Apache 2.0 — reproduce or disprove it on your corpus. The engineering-grade source lives in USECASES.md.

Pick Bernoulli when

  • You need calibrated probabilities for threshold-based auto-decisions
  • Your content is novel, recent, or proprietary (not in public training sets)
  • You ask binary Yes/No questions over semantic content
  • You need multiple questions over the same input (triage, LLM-as-judge, moderation)
  • You need on-prem / in-VPC inference with zero outbound calls

Pick something else when

  • You have 2000+ labeled examples and a fixed label space → BGE-m3 + LR
  • Your task is classical NLP (sentiment, topic, no binary) → DeBERTa-v3-zeroshot
  • You need production-grade safety with high unsafe-recall → Llama Guard / ShieldGemma
  • You need generated text alongside the decision → chat LLM
  • Your answers are open-vocabulary (not a fixed set) → retrieval / NER
When Bernoulli is the right answer

Five scenarios where the benchmarks give us a clear edge.

All numbers from the public benchmarks runs at N=500 (arXiv at N=100 — full available test slice). Qwen2.5-VL-7B throughout.

01

Calibrated probabilities for threshold-based decisions

Across all nine public benchmarks, Bernoulli's NLL is 7–19× better than the same backbone used as a text-and-parse classifier. If your code is if confidence > 0.9: auto_approve, a miscalibrated 0.9 is a product bug — your false-positive rate quietly climbs.

02

Data that wasn't in any public pre-training set

arXiv papers published 2026-09-18 to 2026-10-06 — strictly after both Qwen's and DeBERTa's training cutoffs. Bernoulli 0.95 vs DeBERTa-zeroshot 0.53 — a 42-point gap on content neither model could have memorized. Our architecture generalizes; a zero-shot NLI encoder tuned on fixed data doesn't.

03

Binary Yes/No semantic reasoning

Zero-shot NLI encoders like DeBERTa only score natural-language label hypotheses — they don't handle binary Yes/No. On PAWS (adversarial paraphrase binary): Bernoulli 0.846 / NLL 0.376; BGE-m3+LR crashes to 0.552; DeBERTa n/a by design. For guardrails, policy checks, is-this-X questions — zero-shot encoders are off the map and prompt-and-parse is poorly calibrated.

04

Multiple questions over the same input

Triage, LLM-as-judge, moderation, code review — real workloads need several classifications per input. Prompt-and-parse makes one HTTP call per question; Bernoulli processes the state once and shares it across questions via prefix caching. On A10G 24GB: 1 question in 76 ms, 5 questions in 368 ms, 20 questions in 1466 ms — roughly ~70 ms per additional question.

05

On your own hardware, no API lock-in

Apache 2.0 code, open-weight model, backbone-agnostic (any VL model with a BERNOULLI_MODEL_ID swap). Three deployment modes: self-host in your VPC (flagship), managed-by-us-on-your-cloud (BYOC), or hosted-by-us. The self-host path is zero outbound calls at inference time. If data residency is a hard requirement, we're in a different category from any hosted API.

When Bernoulli is NOT the right answer

Honest: here's where someone else beats us, with the specific alternative to pick.

If you recognize your workload in any of these, pick the named alternative — Bernoulli isn't the right tool for the job.

01

You have 2000+ labeled examples and a fixed label space

Train a classifier. On tasks with labeled data: Banking77 (77-way) BGE+LR 0.95 vs us 0.57, CLINC150-OOS (151-way) BGE+LR 0.80 vs us 0.45, ToxicChat BGE+LR 0.78 vs us 0.69. A trained classifier has fundamentally better signal than any zero-shot approach.

Pick: BGE-m3 + logistic regression
02

Your task is classical NLP where DeBERTa was effectively trained near

DeBERTa-v3-zeroshot was NLI-tuned on datasets covering a lot of academic benchmark content. SST-2 +1.6 pt, AG News +4.6 pt, Banking77 accuracy +6.8 pt. The arXiv post-cutoff crash (0.53) tells you this is training-overlap, not generalization — but if your task is similar, it works.

Pick: DeBERTa-v3-zeroshot
03

Production-grade safety classification

For "flag unsafe prompts" at scale where unsafe-recall is the top metric, specialist safety models are tuned specifically for this. Our XSTest numbers tell the honest story: <1% false-refusal (safe-by-default) but unsafe-recall only 0.28 — we're a first-pass gate, not a final answer.

Pick: Llama Guard 3 / ShieldGemma / WildGuard-7B
04

You need to detect "none of these" on long-tail intent sets

CLINC150-OOS tests OOS detection in a 150-intent space. Our oos_auroc = 0.48 — essentially coin flip. BGE-m3+LR hits 0.78 (not by predicting OOS, but by ranking in-scope confidences well enough that a threshold works). Real limitation of our prompt framing today.

Pick: Embedding classifier + calibrated threshold
05

You need stability over option reorder on >50 options

Our chunked scoring (for choice tasks with >26 options) splits labels into chunks of ≤26 and softmaxes globally. Banking77 reorder stability: 0.07 (DeBERTa and BGE+LR hit 1.00). Architectural, not a bug — cross-chunk debias isn't shipped yet.

Pick: DeBERTa or BGE+LR for >50 options
06

You need generated text, not just a decision

Summaries, explanations, rewritten emails, code suggestions. Bernoulli is for decisions, not content. Use a generative LLM. If you want both a decision and an explanation, run them as two separate calls — one Bernoulli, one chat — and keep the clean separation.

Pick: Generative chat LLM
07

Open-vocabulary answers

Bernoulli assumes your decision has a fixed enumeration of possible answers. If the right answer is "any of these 10,000 product SKUs" or "generate a new SKU name," that's a different tool.

Pick: Retrieval, NER, or text generation
The alternatives

The right tool for each workload, in detail.

If one of these fits better than Bernoulli, use it. We'd rather you solve your problem than use our thing.

DeBERTa-v3-zeroshot

MoritzLaurer/deberta-v3-large-zeroshot-v2.0 · 435M params

NLI-tuned encoder. Treats classification as "entailment of a hypothesis." Choice tasks only; binary Yes/No is unsupported by design.

Pick when Zero-shot classical NLP (sentiment, topic). Choice-only. Single-input-at-a-time. ~37 ms per label on warm GPU.
Skip when Binary Yes/No needed. Post-cutoff content. Multiple questions per input.

BGE-m3 + logistic regression

BAAI/bge-m3 + sklearn.LogisticRegression · ~2.3B params

Sentence embedder + classical ML on top. Needs 500–2000+ labeled examples per task. Wins decisively on long-tail labeled tasks.

Pick when You have labels. Fixed label space. 10+ classes. Latency matters (~12 ms per call on CPU).
Skip when No labels. Changing label space. Limited training data (ECE goes bad fast).

Prompt-and-parse LLM

ChatGPT · Claude · open-weight chat

Ask an LLM in natural language, parse the completion. Current default for "add intelligence to this backend." Terrible calibration (7–19× worse NLL than us).

Pick when You also need generated text (explanation, rewrite, summary). "Right 85% of the time" is enough.
Skip when You need calibrated probabilities. Many questions per input. Can't tolerate parse failures.

Fine-tuned task-specific encoder

BERT · DeBERTa · RoBERTa — your own training

Fine-tune a classifier on your labeled data. Highest accuracy ceiling, biggest ops cost. The "if money and data are no object" answer.

Pick when 10,000+ labeled examples. Stable production task. Accuracy difference matters more than training + maintenance cost.
Skip when Label space shifts. Less data. Training pipeline would be more code than inference.

Specialist safety models

Llama Guard 3 · ShieldGemma · WildGuard-7B

Open-weight models fine-tuned specifically for safety classification. Report token probabilities over safe/unsafe labels — apples-to-apples with Bernoulli when safety is the only question.

Pick when Production safety is the top metric. Training distribution matches your traffic. High unsafe-recall required.
Skip when Safety is one of five questions per input. "Safe-by-default + low over-refusal" matters more than "catch every unsafe case."

Classical ML on hand-engineered features

VADER · regex · lexicons · gradient-boosted trees

Still legitimate in 2026. If your decision reduces to counting words from a known lexicon or matching regex patterns, that's often faster, cheaper, and more auditable than any ML approach.

Pick when Decision reduces to features you can enumerate. Audit trail matters. Latency has to be < 1 ms.
Skip when The semantic distinction is the point (sarcasm, nuance, context).
Honest limitations

What our own benchmarks don't show in our favor.

Equal time to the downsides. Everything here is published in the per-benchmark result files and in the benchmarks README caveats.

Dev-tier backbone throughout

All public numbers are from Qwen2.5-VL-7B. A one-shot H100 rental of the 32B variant lifted the hardest JevBench slice by ~18 points — see the H100 writeup. Expect every row here to improve on the production tier.

Our evaluation, not an independent audit

Every result row carries a reproducibility footer (model revision, dataset hash, seed, hardware) — any number is reproducible or disprovable. The JevBench submission is in the public leaderboard's queue for an external re-run; this page updates once that posts.

No comparison against specialist safety models

WildGuardTest is HF-gated; a Llama Guard 3 vs Bernoulli comparison hasn't shipped. If your workload is production safety where unsafe-recall dominates, we're honestly not pretending to compete with the specialists.

CLINC150-OOS "none of these" is coin flip

oos_auroc = 0.48 at N=500. On long-tail intent sets with an OOS category, our prompt framing doesn't carry enough signal. Investigation-pending, not shipped.

Still not sure if we fit?

Three ways forward: try the live demo, run the benchmarks on your own data, or book a 20-minute fit review with us.