Open-JevHow it was built

What if a model returned decision probabilities directly?

The initial goal was straightforward: after seeing the Jev launch, jevinci, and community game demos, build an open version, starting with Qwen at 2B, 9B, and 27B.

The demos spanned customer support, games, drawing, and agent workflows. They shared a structure: provide the environment state, ask a specific question, and choose from a finite set of actions or levels. Following that structure, we broke each scenario into context, question, candidates, and reference criteria, then connected data generation, training, serving, and replay.

The earliest repository records established typed-decision training and calibration, then added customer support flows, replayable Doom trajectories, Wiki navigation, and local game and workflow control. From the start, each expansion included checks on data splits, input lengths, and checkpoint identity.

Put candidates in the input,
share the scoring head across domains

Customer support might choose between refund, replacement, and escalation; a game might choose left, right, or fire. The candidate set can change without adding a new classification head. Each model size uses one LoRA adapter and one linear head that outputs a scalar; different candidate descriptions produce different representations through the shared backbone.

During training, sources are converted to a common decision-row format while retaining their candidates and target distributions, then mixed and shuffled. Each row computes its loss only over its own candidate set. The first full training used NLL plus a Brier term, followed by temperature fitting on a separate calibration split. Test labels are not used for optimization.

Mechanism and limits: Multidomain training notes · Prefix reuse and numerical checks

What data does it take to support a scenario?

A callable interface is only the starting point. Scenarios need to become training and evaluation units with labels, provenance, splits, and inspectable failures.

The first round prepared tasks around customer support, Doom, and Wiki demos, then added workflows, Snake, tic-tac-toe, runner and platformer games, drawing geometry, and independent reasoning controls. Later expansion covered browser and drone actions, citations, entities, amounts, contact details, context retention, retrieval, and multilingual email handling.

Make the research process open to inspection

The code went into a public repository, data and 2B / 9B model components went to Hugging Face, and the project website brought together results, methods, and videos. Model packages contain LoRA adapters, decision heads, and calibration parameters; running them still requires the pinned Qwen base model.

The demos moved from dense overviews to continuous sequences of saved game states and final goals. The graphics below illustrate this chapter; the actual videos and evidence are in the gallery.

These are verified successful episodes from an early pilot, not game win rates for the later fully trained checkpoints. The public gallery selects successful replays and clearly labels interface walkthroughs; evaluation reports retain failure records. Browser and drone snapshot decisions are also recorded separately from full-task success. Playback duration is not inference latency.

Hard external tasks clarify the next step

We first compared quality and latency on fixed workloads, then added every available public JevBench task. The results show that small open models can already handle many decisions, while harder reasoning still reveals clear gaps.

Scroll horizontally on narrow screens. These results cover 231 public tasks, not the full 534-task benchmark. Open-Jev and Jev return native probabilities; GPT reports probabilities in constrained JSON. Across the 139 Choice tasks, 119 have different candidate orders between the two upstream adapter types. The tasks and scoring protocol are shared, but candidate order and request bytes are not identical.

Efficiency also depends on the workload

The following comes from separate repeated latency experiments: full-response P50 in ms. Each path had 3 warmup requests and 20 measured requests, at concurrency 1.

In these latency charts, 2B runs on one local H100 over loopback HTTP; the other models use hosted HTTPS APIs, so hardware and networks differ. The new 27B JevBench run executed directly across four GPUs on a shared machine. Its timing is excluded from these charts and cannot be interpreted as single-GPU or HTTP/API latency. The 85 ms result does not generalize to every task or establish a speedup on identical hardware. These repeated latency experiments are recorded separately from single-request timings on heterogeneous JevBench tasks. Prefix caching was disabled in all of these Open-Jev comparisons.

Bring harder scenarios into the next training round

The next round expanded data around verified community use cases: customer support, constrained retrieval, contract clauses, RAG evidence, browsers and tools, shell-history selection, game planning, and explicit rubric judgments. The focus was on conditions that change the decision, rather than simply replacing names or candidates.

The community control set adds 51,200 rows: 8 domains and 80 original policy families, drawn from 6,400 base scenarios and 25,600 distinct contexts. Each counterfactual changes just one visible fact and changes the decision. An independent auditor directly parsed the visible rules and checked every label and all 19,200 counterfactual edges.

To complement the structured rule data, we added natural-language inference from WANLI: original premises, hypotheses, and human-reviewed labels were retained and converted into entailment, contradiction, and unknown candidates. The conversion retained 101,207 rows, with 82,045 used for training; connected seed / premise groups from the official test set were isolated in advance.

Training totals 148,639 rows; with each source’s holdouts retained, the full mixture contains 328,672 rows across all splits. Original community data is CC0, WANLI is CC BY 4.0, and the mixture retains source licenses. Different task views in the data remain correlated. Public v1.1 is a redistributable projection of 326,619 rows, including 147,139 training rows. It excludes 2,053 Wiki rows and is not the complete training or evaluation corpus. TypeSafe short questions retain their source-license limitations.

More directions are being prepared: Verified community directions also include skills / model routing, SQL auditing, CMS multilabel classification, and deciding whether approval is needed. Data and held-out evaluations are prepared for these directions, but they still require separate training and audits. They are not verified v1.1 capabilities.

Complete original and expanded holdouts

Old Test / OOD contain 10,532 / 15,920 rows in full; Expanded Test / OOD contain 43,301 / 84,486. Their respective 486 / 474 / 512 / 562 soft-label rows are excluded from hard accuracy but included in probability metrics. The expanded sets preserve the original rows’ inputs, candidate order, and labels exactly. The four columns overlap and must not be added together. Internal evaluation uses a maximum length of 4,096 with no truncation; external JevBench uses a separate 16,384-token protocol.

JevBench and JF100 remain evaluation-only and were not used as source material for this round’s generators. Lexical overlap checks found no matches, but cannot rule out semantic paraphrases or exposure during base-model pretraining. Comparing 27B v1.1 with the Open-Jev-9B changes both model size and earlier training history, so differences cannot be attributed to this new data alone.

The next community dataset is frozen and has passed complete tokenizer checks, but these preparations have no new model results yet. Real-intent routing, executable SQL, approval, and CMS tasks contribute 129,288 new decisions, including 74,921 training rows. With 21,928 earlier-task replay rows, the training set totals 96,849 rows. A separate 1,280-row focused test / OOD evaluation covers 840 original groups, isolated from training, calibration, and validation. Holdouts built from the same synthetic rules may still be optimistic and should be assessed alongside external JevBench results. Internal evaluation method