Benchmark results.
Scope included.

External decision benchmarks

JevBench covers all 231 available public tasks out of 534 total: 72 original, 48 easy and 111 hard. The other 303 private/judge tasks were unavailable. All five streams completed and passed independent replay audits. Native and GPT adapters use different candidate orders on 119 of 139 Choice tasks. Open-Jev and Jev return native probabilities; GPT returns verbalized probability vectors in constrained JSON, not token logprobs. Jev had one vector normalized under the upstream rounding policy (230/231 strict-valid); the other four streams had 231/231 strict-valid vectors.

JF100 has 100 external items in three option rotations: 300 correlated decisions, not 300 independent problems. Jev’s 232/300 is categorical-only accounting and retains two probability-mass flags. The earlier Open-Jev pilot checkpoints are different from the released models; their archived scores do not fill the pending released-model cells.

Retrieval on TREC-DL

TREC uses the downloaded BM25 top 100, nine adaptive 20-passage windows per query and full official qrels with direct-grade gains. Each hosted model completed 97 queries / 873 requests. Strict failures contribute zero over the full 43/54 query denominators. Jev had 108 mass-validation failures affecting 66 queries: 15/43 DL19 and 16/54 DL20 queries were strict-complete. Luna and Astra had no transport or strict-validation failures.

Usage-based standard-list estimates for the complete TREC run: Luna $0.799273; Astra $39.62622. These are estimates, not invoices. Jev cost is unknown; Open-Jev compute is unmetered, not free.

Prepared-data reference matches

Release-v2 evaluates every 26,452 test/OOD row per model: 10,532 test and 15,920 OOD rows. Hard accuracy uses only 10,046 test and 15,446 OOD one-hot targets; the 960 soft-target rows are excluded from these fractions. These are reference matches in prepared task data, not end-to-end workflow success or game win rates. There was no full-data base-model comparison.

The same-76 row is the matched coverage comparison: historical 2B/9B predictions were reused only after exact row and checkpoint checks, adding no new latency. Their additional 64 hard cases remain pending in the broader 140-case suite. A later label audit found equivalent platformer actions and omitted ViZDoom policy constants. The separately labeled post-hoc six-case exclusion gives 2B 60/70, 9B 67/70, Jev 64/70, Luna 57/70 and Astra 69/70; it does not replace the original row.

The IR pilot contains six original query instances and is separate from TREC. FizzBuzz tests integers 1–100 with three typed questions each. Mailroom uses 87 requests, shared families and 921 labeled decisions; 36 inapplicable questions have no gold. These finite controls and correlated views are not representative population or production-success estimates. In these suites GPT emits categorical decisions, not probability vectors.

Latency on repeated workloads

The repeated latency study uses 11 fixed workloads, 20 measured requests and three warmups per workload and provider, concurrency one and no retries. The table shows two workloads; all 11 remain in the linked full report. Open-Jev uses a warm local H100 and loopback HTTP with prefix cache off. Hosted providers use fresh HTTPS connections, including Internet/TLS/routing/scheduling. These are client-observed deployment times, not matched-hardware model speedups, throughput or energy-efficiency measurements. Response validity does not establish equal task quality.

The larger candidate workload is an unfavorable case for uncached Open-Jev 2B. Experimental prefix caching exceeded the probability tolerance on 9/11 workloads despite all 440 paired selected decisions matching; it remains off by default.

Separate JevBench timing diagnostics

JevBench timing is a separate diagnostic: one full-response observation per heterogeneous task, no warmups or retries. It must not be pooled with the repeated-workload latency study. Local model loading is outside the timer; full response completion and validation are inside.

Models, evidence and pending work

• Released Open-Jev 2B: Qwen/Qwen3.5-2B. LoRA + scalar head + saved calibration; base revision 15852e8c16360a2fea060d615a32b45270f8a8fc.
• Released Open-Jev 9B: Qwen/Qwen3.5-9B. LoRA + scalar head + saved calibration; base revision c202236235762e1c871ad0ccb60c8ee5ba337b9a.
• Jev: jev-1.13.0. Recorded hosted version.
• GPT-5.6 Luna: gpt-5.6-luna. Reasoning none; recorded alias, no dated snapshot available.
• GPT-6 Astra: gpt-6-astra. Reasoning low; recorded alias, no dated snapshot available.

The 2B and 9B packages are released LoRA adapters, scalar heads and calibration parameters; pinned base weights and the Open-Jev loader are required. Exact model and package hashes are recorded in the linked reports. Different GPT reasoning settings are not equal budgets.