Bernoulli — Benchmarks

Nine benchmarks.
One thesis: calibration.

Across five academic benchmarks and four use-case benchmarks, Bernoulli hits comparable or better accuracy than the same-backbone generative baseline — and NLL is 7–40× better. Same model, same prompt framing, same examples; the only variable is whether we read the logits at the answer position or let the model generate a token and parse it.

Try it on your own data → Live demo on prepaid HF credits with the exact same backbone and revision as these numbers. Sleeps when idle, ~45 s to wake on first click.
In-house re-run · N=500 · academic 2026-10-15 (+ Lambda A100 for DeBERTa column) · use-case 2026-10-16 · g5.xlarge A10G 24GB · JevBench 7B · 2026-10-08 · JevBench 32B · 2026-10-10 · Lambda H100 80GB
0.95 / 0.53
Bernoulli vs DeBERTa · arXiv post-cutoff

Papers published Sept-Oct 2026, strictly after both models' training cutoffs. DeBERTa's NLI head doesn't generalize; Bernoulli-via-Qwen does.

12×
NLL · PAWS

Adversarial paraphrase binary. Bernoulli 0.85 vs 0.84 accuracy; NLL 0.38 vs 4.53.

0.42
MAE · Yelp 1-5

Lowest rating error of any baseline (Generative 0.46, BGE+LR 0.53). Full distribution, not just argmax.

7-18×
NLL · across 9 benchmarks

Bernoulli vs same-backbone Generative, measured on identical examples. Nine independent receipts.

Academic suite

Six benchmarks that overlap with the Jev paper

Standard open NLP tasks spanning sentiment, topic, intent, emotion, and paraphrase. Bernoulli, same-backbone Generative, and BGE-m3+LR (trained encoder, 500 labeled examples). The Jev-matching set lets our numbers sit next to the category's own.

BenchmarkBaseline AccMacro-F1 ECE (10)BrierNLL ↓ Stability (reorder, reword)
SST-2 · binary sentimentBernoulli0.9140.9140.0240.1360.251(1.00, 0.98)
Generative0.9200.9200.0800.1602.211(0.98, 0.98)
BGE-m3 + LR0.9000.8990.0450.1460.253(1.00, 1.00)
DeBERTa-zeroshot0.9300.9300.0330.1150.219(1.00, 1.00)
AG News · 4-way topic
AG News · 4-way topicBernoulli0.8280.8170.1200.2980.805(0.96, 0.94)
Generative0.7720.7570.2340.4626.167(0.78, 0.88)
BGE-m3 + LR0.8760.8700.1230.2140.430(1.00, 1.00)
DeBERTa-zeroshot0.8740.8680.0790.2210.516(1.00, 1.00)
Banking77 · 77-way intent (chunked)
Banking77 · 77-way intentBernoulli0.5720.1160.1110.5621.401(0.07, 0.68)
Generative0.6400.1130.3480.7089.668(0.26, 0.80)
BGE-m3 + LR0.9500.7290.2360.1510.406(1.00, 1.00)
DeBERTa-zeroshot0.6400.1140.1160.5381.418(1.00, 1.00)
TweetEval-emotion · 4-way
TweetEval-emotion · 4-wayBernoulli0.7840.7440.0840.3010.575(0.89, 0.89)
Generative0.7800.7540.2170.4365.977(0.70, 0.88)
BGE-m3 + LR0.7880.7470.1080.3270.622(1.00, 1.00)
DeBERTa-zeroshot0.7680.6950.0430.3060.584(1.00, 1.00)
PAWS · adversarial paraphrase binary (DeBERTa-zeroshot n/a — BinaryQuestion unsupported by design)
PAWS · adversarial paraphraseBernoulli0.8460.8440.0400.2330.376(n/a, 0.94)
Generative0.8360.8330.1640.3284.532(n/a, 0.93)
BGE-m3 + LR0.5520.5320.0350.4960.690(n/a, 1.00)
arXiv post-cutoff · anti-contamination (papers from Sept-Oct 2026; N=100 — full available test slice)
arXiv post-cutoff · 4-way categoryBernoulli0.9500.9500.0470.1020.207(0.95, 0.94)
Generative0.9200.9190.0800.1602.211(0.86, 0.66)
BGE-m3 + LR0.9400.9400.5090.4530.869(1.00, 1.00)
DeBERTa-zeroshot0.5300.5130.0590.5931.084(1.00, 1.00)

N=500 eval examples per benchmark except arXiv post-cutoff at N=100 (full available test slice) · BGE-m3+LR trained on 2000 labeled examples (random-seeded shuffle for class balance) · stability reorder is n/a for binary (Yes/No swap inverts meaning, not position — see methodology) and for Banking77 reorder=0.07 reflects the chunked-scoring architectural caveat below. DeBERTa stability is (1.00, 1.00) everywhere by architecture — encoder with explicit label hypotheses, no autoregressive position bias. Each benchmark link above opens its full result markdown + JSON sidecar on GitHub.

Use-case suite

Four benchmarks that back the use-case cards

Guardrails, support triage, and content rating — the domains Bernoulli is pitched for. External specialists would be the real comparison targets here (Llama Guard, ShieldGemma, WildGuard) — those columns are planned, not yet wired. For today it's the three in-house baselines, and the calibration gap still holds.

BenchmarkBaseline AccMacro-F1 ECE (10)BrierNLL ↓ Stability (reword)
ToxicChat · toxicity binaryBernoulli0.6900.5500.2210.4870.8000.94
Generative0.6940.5530.3060.6128.460.94
BGE-m3 + LR0.7820.7200.0390.2990.4561.00
XSTest · over-refusal binary
XSTest · over-refusalBernoulli0.6760.6030.1310.4380.6450.73
Generative0.6600.6420.3400.6809.390.66
BGE-m3 + LR— no training split in XSTest (dataset is probe-only, 450 prompts); BGE+LR auto-skipped.
CLINC150-OOS · in-scope accuracy with OOS detection (151-way chunked)
CLINC150-OOS · in-scopeBernoulli0.4480.4180.2800.7281.0280.62
Generative0.2920.2920.7081.41619.560.64
BGE-m3 + LR0.8040.4460.1720.3600.6861.00
Yelp 1-5 stars · rating (5-way ordinal)
Yelp · 1-5 star ratingBernoulli0.6240.6190.1330.5160.8900.80
Generative0.5760.5660.4240.84811.720.69
BGE-m3 + LR0.5720.5730.0820.5470.9721.00
WildGuardTest · response-harm binary (pending auth)
WildGuardTest loader is shipped. The dataset allenai/wildguardmix is gated on Hugging Face; the first run will land once the license is accepted and HF_TOKEN is set on the dev box.

Per-benchmark extras (false-refusal rate on XSTest, OOS AUROC on CLINC150-OOS, MAE / off-by-one on Yelp) are in each benchmark's full result markdown — follow the links above.

External benchmark

JevBench — the Jev-class category's own benchmark

Benchmark Heaven's JevBench measures Jev-class decision models directly: typed answer per task, probabilities over the exact label set, no prose. Bernoulli ran through the official harness at commit bb05a335 against all 231 public tasks. Submission to the official leaderboard at benchmarkheaven.com/jev-models is pending — numbers below are self-reported from the public cohort until the Benchmark Heaven FIFO queue publishes the listing.

Split n Accuracy Brier ECE (10) p50 latency p95 latency
Qwen2.5-VL-7B-Instruct (dev tier — currently serving)
original · standard tier720.9440.1170.05792 ms122 ms
easy481.0000.00020.00392 ms127 ms
hard1110.4230.7360.273338 ms1634 ms
7B weighted2310.706————
Qwen2.5-VL-32B-Instruct (production-size backbone — H100 rental, 2026-10-10)
original · standard tier720.9720.0460.035519 ms568 ms
easy481.0000.00000.001122 ms150 ms
hard1110.6040.5560.189681 ms2124 ms
32B weighted2310.801————

All 231 tasks: operational_success = schema_validity_strict = coverage = 1.000 on both backbones — every task got a response, every response's label set matched the task's exact label set, no distribution needed renormalising. probs_source = "native" throughout. Paraphrase agreement on the standard tier: 0.944 (7B) and 0.986 (32B) over 36 paraphrase pairs. Scaling 7B → 32B lifts weighted accuracy by +9.5 pp and halves Brier on standard + hard tiers. Hard-tier reasoning gains concentrate in multi-step categories (adversarial +50 pp, long_policy / multi_hop / tradeoff +33 pp each). The 32B numbers are a one-shot H100 rental measurement on the base (unquantized) backbone; what we currently serve is 7B. The production target (Qwen2.5-VL-32B-AWQ on an L40S-class GPU) is still awaiting hardware availability. Full writeup: results/2026-10-10-32b.md; human-readable report: reports/2026-10-10-h100-32b.md. The 2026-10-08 7B submission to Benchmark Heaven's FIFO queue is unchanged; 32B goes in as a staged resubmission when the current entry posts.

Methodology

What the numbers actually mean

Every metric answers a specific product-claim question. We run the same backbone (Qwen2.5-VL-7B-Instruct) both ways — Bernoulli reads single-token logits at the answer position; Generative asks in natural language and parses the completion. BGE-m3+LR embeds the input and fits per-task logistic regression on labeled data (500 examples).

Metrics

  • Accuracy, macro-F1 — top-1 label match.
  • ECE (10 bins) — Expected Calibration Error, JevBench convention. Lower is better; 0 is perfect calibration.
  • Brier — mean squared error of top-class probability.
  • NLL — negative log-likelihood of the gold label under the predicted distribution. The headline calibration metric; dramatically worse for the point-mass outputs of a text-and-parse pipeline.
  • Stability (reorder, reword) — fraction of examples whose top-1 is unchanged under option permutations and up to three paraphrased question stems. Binary skips reorder (Yes/No order is semantic, not positional).

Baselines

  • Bernoulli over HTTP against the running /v1/decide server. Reverse debias on choice; no debias on binary (Yes/No are semantic — reversing their order changes meaning, not position).
  • Generative over HTTP against the same server's /v1/generate. Same backbone, same prompt framing, parsed completion → point mass on parsed answer or uniform on parse failure.
  • BGE-m3 + LR — embed inputs with BAAI/bge-m3 (CLS pooling, L2-normalise), train scikit-learn's LogisticRegression on 2000 labeled training examples (seeded shuffle for class balance). Honest baseline for "could we have used embeddings?"
  • DeBERTa-v3-zeroshot — MoritzLaurer/deberta-v3-large-zeroshot-v2.0. The strongest open zero-shot encoder — NLI-tuned, which means it only scores ChoiceQuestion (natural-language label hypotheses); binary Yes/No tasks like PAWS are n/a by design. Produced on a one-shot Lambda A100 (sidesteps evicting the Bernoulli vLLM server on the dev box).
Honest caveats

What this run is not

Four things worth naming up front — the full engineering caveat list, with code references and per-benchmark footnotes, lives in benchmarks/README.md on GitHub.

  • Our evaluation, not an independent audit. Every result row carries a reproducibility footer (model revision, dataset hash, seed, hardware) so any number here can be reproduced or disproved. The JevBench submission is in the public leaderboard's queue for an external re-run; this page updates once that posts.
  • Dev-tier backbone. All numbers are from a 7B open-weight model on a single commodity GPU. A one-shot rental on a 32B production backbone (full writeup) lifted the hardest JevBench slice by ~18 points — expect every row here to improve on the production tier.
  • Not every baseline fits every task. DeBERTa-zeroshot is natural-language-inference-tuned so it only scores choice tasks (binary Yes/No like PAWS is n/a by design). Banking77's 77-way label space exceeds our single-pass alphabet, so stability under option reorder reads low on that one benchmark — architectural, documented per-benchmark.
  • CLINC150-OOS oos_auroc ≈ 0.48 for us. On the "none of these" probability story, we're at coin flip. BGE-m3+LR hits 0.78 — not by predicting OOS correctly (it never does, oos_recall 0) but because its softmax over the 150 in-scope classes carries enough signal for a well-ranked threshold. Likely prompt-framing + stem rewording on our side; investigation pending.
Reproducibility

Pinned so you can audit, rerun, or disprove

Every row above is backed by a result file on GitHub with the exact model revision, dataset hash, random seed, and hardware so you can reproduce or disprove any number.

ModelQwen/Qwen2.5-VL-7B-Instruct @ cc59489
HardwareAWS g5.xlarge · A10G 24GB · us-east-1a
TransportHTTP to the running Bernoulli server
Sample size500 eval per benchmark (arXiv at 100 — full available slice) · 2000 training examples for the trained-encoder baseline
Date2026-10-15 (academic) · 2026-10-16 (use-cases)

To rerun on your own backbone or hardware, clone the repo and follow the benchmarks guide on GitHub — one command per benchmark, writes a timestamped result file.

Try it against your own data

Open weights, runs on-prem, benchmarks reproducible from the repo. Grab the codebase, or book a demo and we'll walk through what it would mean to run on your corpus.

View on GitHub