Across five academic benchmarks and four use-case benchmarks, Bernoulli hits comparable or better accuracy than the same-backbone generative baseline — and NLL is 7–40× better. Same model, same prompt framing, same examples; the only variable is whether we read the logits at the answer position or let the model generate a token and parse it.
Papers published Sept-Oct 2026, strictly after both models' training cutoffs. DeBERTa's NLI head doesn't generalize; Bernoulli-via-Qwen does.
Adversarial paraphrase binary. Bernoulli 0.85 vs 0.84 accuracy; NLL 0.38 vs 4.53.
Lowest rating error of any baseline (Generative 0.46, BGE+LR 0.53). Full distribution, not just argmax.
Bernoulli vs same-backbone Generative, measured on identical examples. Nine independent receipts.
Standard open NLP tasks spanning sentiment, topic, intent, emotion, and paraphrase. Bernoulli, same-backbone Generative, and BGE-m3+LR (trained encoder, 500 labeled examples). The Jev-matching set lets our numbers sit next to the category's own.
| Benchmark | Baseline | Acc | Macro-F1 | ECE (10) | Brier | NLL ↓ | Stability (reorder, reword) |
|---|---|---|---|---|---|---|---|
| SST-2 · binary sentiment | Bernoulli | 0.914 | 0.914 | 0.024 | 0.136 | 0.251 | (1.00, 0.98) |
| Generative | 0.920 | 0.920 | 0.080 | 0.160 | 2.211 | (0.98, 0.98) | |
| BGE-m3 + LR | 0.900 | 0.899 | 0.045 | 0.146 | 0.253 | (1.00, 1.00) | |
| DeBERTa-zeroshot | 0.930 | 0.930 | 0.033 | 0.115 | 0.219 | (1.00, 1.00) | |
| AG News · 4-way topic | |||||||
| AG News · 4-way topic | Bernoulli | 0.828 | 0.817 | 0.120 | 0.298 | 0.805 | (0.96, 0.94) |
| Generative | 0.772 | 0.757 | 0.234 | 0.462 | 6.167 | (0.78, 0.88) | |
| BGE-m3 + LR | 0.876 | 0.870 | 0.123 | 0.214 | 0.430 | (1.00, 1.00) | |
| DeBERTa-zeroshot | 0.874 | 0.868 | 0.079 | 0.221 | 0.516 | (1.00, 1.00) | |
| Banking77 · 77-way intent (chunked) | |||||||
| Banking77 · 77-way intent | Bernoulli | 0.572 | 0.116 | 0.111 | 0.562 | 1.401 | (0.07, 0.68) |
| Generative | 0.640 | 0.113 | 0.348 | 0.708 | 9.668 | (0.26, 0.80) | |
| BGE-m3 + LR | 0.950 | 0.729 | 0.236 | 0.151 | 0.406 | (1.00, 1.00) | |
| DeBERTa-zeroshot | 0.640 | 0.114 | 0.116 | 0.538 | 1.418 | (1.00, 1.00) | |
| TweetEval-emotion · 4-way | |||||||
| TweetEval-emotion · 4-way | Bernoulli | 0.784 | 0.744 | 0.084 | 0.301 | 0.575 | (0.89, 0.89) |
| Generative | 0.780 | 0.754 | 0.217 | 0.436 | 5.977 | (0.70, 0.88) | |
| BGE-m3 + LR | 0.788 | 0.747 | 0.108 | 0.327 | 0.622 | (1.00, 1.00) | |
| DeBERTa-zeroshot | 0.768 | 0.695 | 0.043 | 0.306 | 0.584 | (1.00, 1.00) | |
| PAWS · adversarial paraphrase binary (DeBERTa-zeroshot n/a — BinaryQuestion unsupported by design) | |||||||
| PAWS · adversarial paraphrase | Bernoulli | 0.846 | 0.844 | 0.040 | 0.233 | 0.376 | (n/a, 0.94) |
| Generative | 0.836 | 0.833 | 0.164 | 0.328 | 4.532 | (n/a, 0.93) | |
| BGE-m3 + LR | 0.552 | 0.532 | 0.035 | 0.496 | 0.690 | (n/a, 1.00) | |
| arXiv post-cutoff · anti-contamination (papers from Sept-Oct 2026; N=100 — full available test slice) | |||||||
| arXiv post-cutoff · 4-way category | Bernoulli | 0.950 | 0.950 | 0.047 | 0.102 | 0.207 | (0.95, 0.94) |
| Generative | 0.920 | 0.919 | 0.080 | 0.160 | 2.211 | (0.86, 0.66) | |
| BGE-m3 + LR | 0.940 | 0.940 | 0.509 | 0.453 | 0.869 | (1.00, 1.00) | |
| DeBERTa-zeroshot | 0.530 | 0.513 | 0.059 | 0.593 | 1.084 | (1.00, 1.00) | |
N=500 eval examples per benchmark except arXiv post-cutoff at N=100 (full available test slice) · BGE-m3+LR trained on 2000 labeled examples (random-seeded shuffle for class balance) · stability reorder is n/a for binary (Yes/No swap inverts meaning, not position — see methodology) and for Banking77 reorder=0.07 reflects the chunked-scoring architectural caveat below. DeBERTa stability is (1.00, 1.00) everywhere by architecture — encoder with explicit label hypotheses, no autoregressive position bias. Each benchmark link above opens its full result markdown + JSON sidecar on GitHub.
Guardrails, support triage, and content rating — the domains Bernoulli is pitched for. External specialists would be the real comparison targets here (Llama Guard, ShieldGemma, WildGuard) — those columns are planned, not yet wired. For today it's the three in-house baselines, and the calibration gap still holds.
| Benchmark | Baseline | Acc | Macro-F1 | ECE (10) | Brier | NLL ↓ | Stability (reword) |
|---|---|---|---|---|---|---|---|
| ToxicChat · toxicity binary | Bernoulli | 0.690 | 0.550 | 0.221 | 0.487 | 0.800 | 0.94 |
| Generative | 0.694 | 0.553 | 0.306 | 0.612 | 8.46 | 0.94 | |
| BGE-m3 + LR | 0.782 | 0.720 | 0.039 | 0.299 | 0.456 | 1.00 | |
| XSTest · over-refusal binary | |||||||
| XSTest · over-refusal | Bernoulli | 0.676 | 0.603 | 0.131 | 0.438 | 0.645 | 0.73 |
| Generative | 0.660 | 0.642 | 0.340 | 0.680 | 9.39 | 0.66 | |
| BGE-m3 + LR | — no training split in XSTest (dataset is probe-only, 450 prompts); BGE+LR auto-skipped. | ||||||
| CLINC150-OOS · in-scope accuracy with OOS detection (151-way chunked) | |||||||
| CLINC150-OOS · in-scope | Bernoulli | 0.448 | 0.418 | 0.280 | 0.728 | 1.028 | 0.62 |
| Generative | 0.292 | 0.292 | 0.708 | 1.416 | 19.56 | 0.64 | |
| BGE-m3 + LR | 0.804 | 0.446 | 0.172 | 0.360 | 0.686 | 1.00 | |
| Yelp 1-5 stars · rating (5-way ordinal) | |||||||
| Yelp · 1-5 star rating | Bernoulli | 0.624 | 0.619 | 0.133 | 0.516 | 0.890 | 0.80 |
| Generative | 0.576 | 0.566 | 0.424 | 0.848 | 11.72 | 0.69 | |
| BGE-m3 + LR | 0.572 | 0.573 | 0.082 | 0.547 | 0.972 | 1.00 | |
| WildGuardTest · response-harm binary (pending auth) | |||||||
WildGuardTest loader is shipped. The dataset allenai/wildguardmix is gated on Hugging Face; the first run will land once the license is accepted and HF_TOKEN is set on the dev box. | |||||||
Per-benchmark extras (false-refusal rate on XSTest, OOS AUROC on CLINC150-OOS, MAE / off-by-one on Yelp) are in each benchmark's full result markdown — follow the links above.
Benchmark Heaven's JevBench measures Jev-class decision models directly: typed answer per task, probabilities over the exact label set, no prose. Bernoulli ran through the official harness at commit bb05a335 against all 231 public tasks. Submission to the official leaderboard at benchmarkheaven.com/jev-models is pending — numbers below are self-reported from the public cohort until the Benchmark Heaven FIFO queue publishes the listing.
| Split | n | Accuracy | Brier | ECE (10) | p50 latency | p95 latency |
|---|---|---|---|---|---|---|
| Qwen2.5-VL-7B-Instruct (dev tier — currently serving) | ||||||
| original · standard tier | 72 | 0.944 | 0.117 | 0.057 | 92 ms | 122 ms |
| easy | 48 | 1.000 | 0.0002 | 0.003 | 92 ms | 127 ms |
| hard | 111 | 0.423 | 0.736 | 0.273 | 338 ms | 1634 ms |
| 7B weighted | 231 | 0.706 | — | — | — | — |
| Qwen2.5-VL-32B-Instruct (production-size backbone — H100 rental, 2026-10-10) | ||||||
| original · standard tier | 72 | 0.972 | 0.046 | 0.035 | 519 ms | 568 ms |
| easy | 48 | 1.000 | 0.0000 | 0.001 | 122 ms | 150 ms |
| hard | 111 | 0.604 | 0.556 | 0.189 | 681 ms | 2124 ms |
| 32B weighted | 231 | 0.801 | — | — | — | — |
All 231 tasks: operational_success = schema_validity_strict = coverage = 1.000 on both backbones — every task got a response, every response's label set matched the task's exact label set, no distribution needed renormalising. probs_source = "native" throughout. Paraphrase agreement on the standard tier: 0.944 (7B) and 0.986 (32B) over 36 paraphrase pairs. Scaling 7B → 32B lifts weighted accuracy by +9.5 pp and halves Brier on standard + hard tiers. Hard-tier reasoning gains concentrate in multi-step categories (adversarial +50 pp, long_policy / multi_hop / tradeoff +33 pp each). The 32B numbers are a one-shot H100 rental measurement on the base (unquantized) backbone; what we currently serve is 7B. The production target (Qwen2.5-VL-32B-AWQ on an L40S-class GPU) is still awaiting hardware availability. Full writeup: results/2026-10-10-32b.md; human-readable report: reports/2026-10-10-h100-32b.md. The 2026-10-08 7B submission to Benchmark Heaven's FIFO queue is unchanged; 32B goes in as a staged resubmission when the current entry posts.
Every metric answers a specific product-claim question. We run the same backbone (Qwen2.5-VL-7B-Instruct) both ways — Bernoulli reads single-token logits at the answer position; Generative asks in natural language and parses the completion. BGE-m3+LR embeds the input and fits per-task logistic regression on labeled data (500 examples).
/v1/decide server. Reverse debias on choice; no debias on binary (Yes/No are semantic — reversing their order changes meaning, not position)./v1/generate. Same backbone, same prompt framing, parsed completion → point mass on parsed answer or uniform on parse failure.BAAI/bge-m3 (CLS pooling, L2-normalise), train scikit-learn's LogisticRegression on 2000 labeled training examples (seeded shuffle for class balance). Honest baseline for "could we have used embeddings?"MoritzLaurer/deberta-v3-large-zeroshot-v2.0. The strongest open zero-shot encoder — NLI-tuned, which means it only scores ChoiceQuestion (natural-language label hypotheses); binary Yes/No tasks like PAWS are n/a by design. Produced on a one-shot Lambda A100 (sidesteps evicting the Bernoulli vLLM server on the dev box).Four things worth naming up front — the full engineering caveat list, with code references and per-benchmark footnotes, lives in benchmarks/README.md on GitHub.
n/a by design). Banking77's 77-way label space exceeds our single-pass alphabet, so stability under option reorder reads low on that one benchmark — architectural, documented per-benchmark.oos_auroc ≈ 0.48 for us. On the "none of these" probability story, we're at coin flip. BGE-m3+LR hits 0.78 — not by predicting OOS correctly (it never does, oos_recall 0) but because its softmax over the 150 in-scope classes carries enough signal for a well-ranked threshold. Likely prompt-framing + stem rewording on our side; investigation pending.Every row above is backed by a result file on GitHub with the exact model revision, dataset hash, random seed, and hardware so you can reproduce or disprove any number.
To rerun on your own backbone or hardware, clone the repo and follow the benchmarks guide on GitHub — one command per benchmark, writes a timestamped result file.
Open weights, runs on-prem, benchmarks reproducible from the repo. Grab the codebase, or book a demo and we'll walk through what it would mean to run on your corpus.