Benchmark results.
Scope included.

Keep benchmark versions separate.

The 231-public tables below use historical JevBench commit f8ce713. Newer independent evaluations have different tasks and metrics. The v1.4 report included 29.9% sealed versus 77.5% public accuracy for 9B; its sealed tasks had a different difficulty mixture. This remains a historical record, not the current result.

Competence is chance-corrected, not accuracy. v1.5 matches open/sealed tier mixtures. Noul probabilities between 0.2 and 0.8 abstain and count wrong; reported coverage and calibration matter. Local leaderboard timing includes a synthetic transport adjustment and cost estimates, separate from raw measurements.

Fresh independent Open-Jev-27B-v1.1: pending. A pinned install/serve/checksum handoff is available for an evaluator-owned host. Preparing it or rerunning old public tasks does not create a sealed result. Aggregate results and a run manifest are sufficient; private questions or labels are not requested.

Efficiency by model and hardware.

The whole-model differences are small and do not establish statistically significant gains. Sequential model runs were not interleaved. Both paths reached the same 6.917 GiB maximum per-request peak allocation. The operator ratio does not describe whole-model or HTTP performance.

All 168 in-process and 168 HTTP parity comparisons passed on the recorded workloads: maximum probability error 1e-7, zero decision changes and zero audited threshold flips. The isolated operator separately passed all nine cases with maximum logit error 1.12e-7. Checkpoint, base/tokenizer files and source identities were verified. The Python 3.12 runtime used existing runtime312 packages plus a selected dependency overlay; this was not a fresh clean Linux installation.

Released 27B on H100

Torch and Triton-tail completed eight workloads with 168 in-process and 168 loopback HTTP parity checks. Maximum probability error was 1.89e-6, with zero decision or audited threshold changes. Both reached 54.124 GiB request peak allocation. Warm P50 ratios were 1.004–1.020× in-process and 1.002–1.029× over HTTP. Sequential runs and a short concurrent 2B pilot on another GPU prevent a statistically significant speedup claim.

The community fast-CUDA backend executed the released 27B but failed the unchanged probability gate: 0.00593 versus 0.0001. Its one-sample diagnostic latency is not an accepted speedup. The original training queue resumed its model, optimizer, sampler tree and actual sampling after the allocation.

One fixed synthetic training pilot

Released 2B completed 64 fixed steps on original counterfactual controls. Test argmax accuracy improved 84/128 (65.625%) → 99/128 (77.34375%); controlled OOD improved 83/128 (64.84375%) → 95/128 (74.21875%). Numeric Test stayed 30% → 30%, numeric OOD fell 35% → 30%, and timeline OOD fell 58.33% → 54.17%. Saved/reloaded probability error was zero. The adapted checkpoint is experimental and is not a released model. This is one synthetic pilot seed; temperature was fitted only on separate synthetic calibration rows.

Temporal-window follow-up: aggregate gain, worse exclusions

A separate fixed 64-step 2B adaptation improved synthetic Test 15/32 → 24/32 and OOD 17/32 → 25/32. Outside-window decisions regressed: Test 2/8 → 0/8 and OOD 5/8 → 1/8. All 12 outside-window Choice cases selected accept although reject was correct. Old v4 regression scores also fell. The adapted checkpoint has not replaced released weights. Inspect subgroup, temperature and regression results ↗

Frozen 64-task public-development comparison

No Choice or Noul accuracy gain was observed: Choice stayed 17/34 and Noul 9/23. All Choice decisions were unchanged. Noul answered coverage changed 17/23 → 18/23; the extra answer was incorrect. Across seven Score items, expected-position normalized MAE fell 0.22393 → 0.20645; auxiliary argmax correct stayed 2/7. Four Score items improved in MAE and three regressed. Equal-type, available-tier-weighted subset competence changed 22.835 → 24.257, driven only by Score.

The 16,384-token limit, source and base revision matched. Checkpoint weights and saved temperature changed together, so this does not isolate weight adaptation from recalibration. This is one development sample from 231 accessible public items, with no uncertainty estimate or official I_open, blind/sealed, composite or rank claim. Natural-support improvement is not established. Full 9B/27B inference on this H200 remains pending; the separate H100 27B validation above records both passed and failed backends.

A local support trial, with a negative baseline result.

On 256 natural BANKING77 official-test utterances, released Open-Jev-2B routed 166/256 correctly (64.8%), versus 211/256 (82.4%) for retrieval top-1 BM25. Candidate recall was 249/256 (97.3%). The frozen calibration policy accepted 88/256 rows (34.4% coverage), with 14/88 wrong (15.9% accepted error); 168/256 require review. Its 90% accepted-accuracy calibration target did not transfer to test. No test threshold was fitted afterwards.

The workflow retrieves eight intents plus a review option, allows named human corrections and exports CSV. An actual-model browser check exercised 12 natural training examples, all left unresolved for review, with zero human corrections and zero completed external human trials. It requires your own trained checkpoint server; the static website does not provide a live GPU trial. The model ranker needs improvement before replacing the stronger retrieval baseline.

Five public community excerpts motivate these workflow tests. They describe TypeSafe Jev and related experiments, not Open-Jev customer endorsements. Prior BANKING77 training exposure is possible; this is a supervised routing sample, not an untouched-task zero-shot or OOS benchmark.

Full internal evaluation

27B v1.1 completed all 127,787 expanded held-out rows, with zero failures, missing rows or duplicates. Old Test/OOD cover all 10,532 / 15,920 rows; Expanded Test/OOD cover all 43,301 / 84,486. The 486 / 474 / 512 / 562 soft-target rows respectively remain in probability metrics. Original panels are unchanged-content subsets of the expanded panels and overlap.

Open-Jev-2B / Open-Jev-9B expanded evaluations have not run. New 2B training stopped; new 9B did not start. Internal max length is 4,096 without truncation, saved temperature, row batch size one and cache off. JevBench uses a separate 16,384-token protocol. Shared multi-GPU timing is not single-GPU or HTTP/API latency.

External decision benchmarks

JevBench covers all 231 available public tasks out of 534 total: 72 original, 48 easy and 111 hard. The other 303 private/judge tasks were unavailable. All six streams completed and passed independent replay audits. Native and GPT adapters use different candidate orders on 119 of 139 Choice tasks. Open-Jev and Jev return native probabilities; GPT returns verbalized probability vectors in constrained JSON, not token logprobs. Jev had one vector normalized under the upstream rounding policy (230/231 strict-valid); the other five streams had 231/231 strict-valid vectors.

27B v1.1 scores 197/231 (85.28%) overall and 80/111 (72.07%) Hard, above Open-Jev-2B / Open-Jev-9B and still three overall and one Hard answer behind Jev. Model scale, data and prior training all differ. The 27B direct-execution timings are not included in the HTTP/HTTPS comparison.

JF100 has 100 external items in three option rotations: 300 correlated decisions, not 300 independent problems. Jev’s 232/300 is categorical-only accounting and retains two probability-mass flags. The earlier Open-Jev pilot checkpoints are different from the released models; their archived scores do not fill the pending released-model cells.

Retrieval on TREC-DL

TREC uses the downloaded BM25 top 100, nine adaptive 20-passage windows per query and full official qrels with direct-grade gains. Each hosted model completed 97 queries / 873 requests. Strict failures contribute zero over the full 43/54 query denominators. Jev had 108 mass-validation failures affecting 66 queries: 15/43 DL19 and 16/54 DL20 queries were strict-complete. Luna and Astra had no transport or strict-validation failures.

Usage-based standard-list estimates for the complete TREC run: Luna $0.799273; Astra $39.62622. These are estimates, not invoices. Jev cost is unknown; Open-Jev compute is unmetered, not free.

Prepared-data reference matches

Release-v2 evaluates every 26,452 test/OOD row per model: 10,532 test and 15,920 OOD rows. Hard accuracy uses only 10,046 test and 15,446 OOD one-hot targets; the 960 soft-target rows are excluded from these fractions. These are reference matches in prepared task data, not end-to-end workflow success or game win rates. There was no full-data base-model comparison.

The same-76 row is the matched coverage comparison: historical 2B/9B predictions were reused only after exact row and checkpoint checks, adding no new latency. Their additional 64 hard cases remain pending in the broader 140-case suite. A later label audit found equivalent platformer actions and omitted ViZDoom policy constants. The separately labeled post-hoc six-case exclusion gives 2B 60/70, 9B 67/70, Jev 64/70, Luna 57/70 and Astra 69/70; it does not replace the original row.

The IR pilot contains six original query instances and is separate from TREC. FizzBuzz tests integers 1–100 with three typed questions each. Mailroom uses 87 requests, shared families and 921 labeled decisions; 36 inapplicable questions have no gold. These finite controls and correlated views are not representative population or production-success estimates. In these suites GPT emits categorical decisions, not probability vectors.

Latency on repeated workloads

The repeated latency study uses 11 fixed workloads, 20 measured requests and three warmups per workload and provider, concurrency one and no retries. The table shows two workloads; all 11 remain in the linked full report. Open-Jev uses a warm local H100 and loopback HTTP with prefix cache off. Hosted providers use fresh HTTPS connections, including Internet/TLS/routing/scheduling. These are client-observed deployment times, not matched-hardware model speedups, throughput or energy-efficiency measurements. Response validity does not establish equal task quality.

The larger candidate workload is an unfavorable case for uncached Open-Jev-2B. Experimental prefix caching exceeded the probability tolerance on 9/11 workloads despite all 440 paired selected decisions matching; it remains off by default.

Separate JevBench timing diagnostics

JevBench timing is a separate diagnostic: one full-response observation per heterogeneous task, no warmups or retries. It must not be pooled with the repeated-workload latency study. Local model loading is outside the timer; full response completion and validation are inside.

Models, evidence and pending work

• Open-Jev-2B: Qwen/Qwen3.5-2B. LoRA + scalar head + saved calibration; base revision 15852e8c16360a2fea060d615a32b45270f8a8fc.
• Open-Jev-9B: Qwen/Qwen3.5-9B. LoRA + scalar head + saved calibration; base revision c202236235762e1c871ad0ccb60c8ee5ba337b9a.
• Open-Jev 27B v1.1: Qwen/Qwen3.8-27B. LoRA rank 8 + FP32 scalar head + saved temperature 2.5343690298472983; base revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0. Checkpoint tree SHA-256 c49994563c3c4f04a99d9130203c4e526f4ae5086c84deec57698d18cb652e71.
• Jev: jev-1.13.0. Recorded hosted version.
• GPT-5.6 Luna: gpt-5.6-luna. Reasoning none; recorded alias, no dated snapshot available.
• GPT-6 Astra: gpt-6-astra. Reasoning low; recorded alias, no dated snapshot available.

The 2B, 9B and 27B v1.1 packages contain LoRA adapters, scalar heads and calibration parameters; pinned base weights and the Open-Jev loader are required. Exact model and package hashes are recorded in the linked reports. Different GPT reasoning settings are not equal budgets.