QWEN3.6-27B · MATH · R-KV 128
−63.90 percentage points
Full-dataset accuracy falls from 72.40% with FullKV to 8.50%. The accuracy loss at this budget is substantial; the scores alone do not identify its mechanism.
R-KV · Full benchmark audit
Full-dataset evaluation
Progress and results
5 models, 4 KV configurations, and the complete GSM8K and MATH test sets. This report combines current scores, scoring coverage, and remote job status, with the calculation basis for every result.
Action needed: resume the remaining 5 shards for Qwen3.5-0.8B. On the evening of October 10, the original queues detected other compute processes on the target GPUs. The idle-GPU check caused the controllers to exit. Ranks 0/1/2 are complete; ranks 3–7 have not started. View queue interruption evidence and next steps
01 / RESULTS
Every configuration, in detail
Score = correct answers ÷ scored questions. Each result shows both generated and scored counts, so generation progress is distinct from completion.
Full dataset scored * Partial-sample score / length — No score yet
Scroll horizontally · Open “Details” for correct counts, output length, output-cap hits, and compression triggers | Model | KV configuration | GSM8K generated / scored Full dataset: 1,319 questions | GSM8K score Correct / scored | GSM8K FullKV avg output tokens | MATH generated / scored Full dataset: 5,000 questions | MATH score Correct / scored | MATH FullKV avg output tokens | Configuration status Based on both test sets | Details |
|---|
02 / COMPARISON
Read scores alongside coverage
Compare budgets within the same model. Changes from FullKV are shown in percentage points only when both configurations have completed the dataset.
KV budget and accuracy
Fixed 0–100% scaleSolid bars indicate full-dataset results; hatched bars indicate partial samples.
0255075100%
Evaluation progress by model
25,276 evaluations per model4 configurations × (1,319 GSM8K questions + 5,000 MATH questions).
Scored Generated, awaiting scoring Not yet generated
A total of 2,259 generated records await scoring, and another 41,900 question evaluations await generation. Each model and KV configuration counts as a separate evaluation.
GEMMA4-E2B · MATH · R-KV 512
−11.52 percentage points
GSM8K scores 89.92%, close to FullKV’s 90.07%; MATH scores 75.78%, below FullKV’s 87.30%. The two tasks differ in their sensitivity to compression.
GEMMA4-E2B · MATH · R-KV 1024
−2.00 percentage points
MATH scores 85.30% and GSM8K scores 90.07%. Among the completed Gemma4 compression configurations, this one retains accuracy best.
03 / OPERATIONS
Generation continues; one set of queues needs resuming
Job status comes from remote read-only checks performed in this session around 12:08 PDT on October 13. This report does not start, stop, or reschedule jobs.
Key events and queue interruption
Qwen0.8 awaiting restart Qwen and Gemma hybrid-attention adaptation completed
Real model weights passed native token parity and full-attention-only compression checks. Native linear state / SWA is preserved; Gemma4 shared KV is compressed at the producing layer. The session records report 58 CPU checks passed, with code committed and pushed.
Scaled to 18 generation workers; first configuration completed
Qwen27 FullKV was the complete result at that point. The main session and old local report have not been updated since. Independent remote processes can keep generating, so the old report does not reflect current progress.
Strict idle-GPU checks failed before the next shards started
After rank2 / rank1 finished, N4-1 GPU7 / GPU6 respectively detected other compute processes and raised “Target card is not strictly idle.” The controllers exited with returncode=1 before launching the next child process. There was no automatic wait-and-retry for an idle GPU.
7 configurations complete; 10 generation workers still running
Gemma4 has completed every configuration. Qwen27 is continuing MATH for R-KV 512. Later Gemma3-270M and 1B shards have moved to N1-1 and continue producing output. The Gemma queues on the original nodes completed and exited normally.
Remaining Qwen0.8 queues
Ranks 0, 1, and 2 are complete. The original GPU6 queue was [1, 3, 5, 7], and the GPU7 queue was [2, 4, 6]. Ranks 3–7 have not started; resume them after rechecking GPU availability and ensuring each shard has a single owner.
Ranks 0, 1, and 2 are complete. The original GPU6 queue was [1, 3, 5, 7], and the GPU7 queue was [2, 4, 6]. Ranks 3–7 have not started; resume them after rechecking GPU availability and ensuring each shard has a single owner.
Observed generation processes by node
10 generation workers| Node | Generation jobs for this project | Count |
|---|---|---|
| N1-1 | Qwen27 × 2 Gemma270M × 1 · Gemma1B × 1 | 4 |
| N1-3 | Qwen27 × 6 | 6 |
| N4-1 | No generation processes found for this project | 0 |
| N4-2 | No generation processes found for this project | 0 |
These counts include generation child processes, exclude queue controllers, and do not indicate that a node is idle. The decline from 18 to 10 workers reflects completed jobs, queue migration, and the interrupted Qwen0.8 queues.
Actual status of automatic follow-up
The local configuration is still marked ACTIVE, but the last session reply was on October 7, and the Paseo query returned no listed schedules. The available evidence does not confirm that automatic follow-up is currently running as intended.
04 / PROTOCOL & EVIDENCE
Protocol and verification evidence
The score denominator, completion criteria, and limits of the conclusions supported by this snapshot.
Fixed generation protocol
- Models and configurations
- 5 models × FullKV / R-KV 128 / 512 / 1024
- Evaluation datasets
- GSM8K test: 1,319; MATH test: 5,000. Neither MATH-500 nor pilot subsets were used as substitutes.
- Inference settings
- BF16 · greedy · batch = 1 · seed = 42
- Attention
- Native SDPA; cuDNN SDPA disabled
- Compression scope
- No compression during prefill. Full attention only, compressed every 128 decode steps. Native SWA and linear state are preserved.
- Adaptation parameters
- window_size = 8 · mix_lambda = 0.1 · retain_ratio = 0.2
- Output limit
- GSM8K 8,192 tokens；MATH 16,384 tokens
- Lengths retrieved at
- FullKV output length
- Actual generated continuation tokens, excluding the prompt and including thinking and generated special / EOS tokens, not just the final answer text.
- FullKV length statistics
- Covers all generated FullKV samples in the snapshot, including unscored samples. For even n, the median averages the middle pair; P95 uses the nearest-rank value at sorted position ceil(0.95 × n).
- Thinking
- Enabled for Qwen3.6-27B; disabled for Qwen0.8 / Gemma3 / Gemma4. Compare configurations within the same model first.
- Sharding and merging
- Each model uses 8 fixed shards. Results are merged across nodes by question ID, with source fingerprints verified.
How to read this report
Accuracy = correct answers ÷ scored questions × 100%
- Dataset completion and configuration completion differ. A dataset result is complete once every GSM8K or MATH question has been generated and scored. A configuration is complete only when both datasets are finished. This snapshot has 16/40 dataset results and 7/20 configurations complete .
- Partial scores may cover different questions. Scores marked * cover only the samples scored so far. Gemma1B R-KV1024’s partial score of 55.93% cannot be directly interpreted as an improvement over FullKV’s full-dataset score of 55.19%.
- Generation and scoring can progress separately. A total of 2,259 records await scoring. Results without scores show “Awaiting scoring”; those without output show “Not started.”
- Questions that reach the output limit remain in the test. An output-cap hit means the output token count reaches the cap, including cases where EOS lands exactly at the limit. Its association with accuracy changes does not establish causation.
- Compression triggers do not measure GPU memory savings. “Samples triggering compression” counts generated samples with at least one eviction. It is neither the fraction of tokens removed nor a measure of throughput or GPU memory gains.
- This report makes no claims about performance gains. This snapshot lacks comparable latency, throughput, and peak GPU memory measurements from exclusive GPU runs, so it cannot quantify speed or memory gains.
Expand sources, versions, and audit records
4-node aggregation · 0 verification errors · Fixed generation protocol and dataset SHA256
Expand sources, versions, and audit records
4-node aggregation · 0 verification errors · Fixed generation protocol and dataset SHA256
Source data and pinned versions
Code branch: codex/hybrid-full-attention-math-20261001 ↗bdff2101133f97896bae84fcbb1dc1468268cf5d
Session and observation scope
Main session: R-KV Qwen
Sampled at 2026-10-13 19:07:39 UTC. Results were aggregated through read-only checks on N4-2, N4-1, N1-1, and N1-3. Existing records passed checks for models, generation parameters, actual dataset hashes, question-source fingerprints, and cross-node duplicates. Complete groups have IDs that exactly cover the expected range. Coverage counts show the missing questions in incomplete groups.
The 10-worker count comes from follow-up process checks during the same query. The initial N1-1 process probe timed out; subsequent read-only ps checks and output-file timestamps confirmed 4 generation processes. Old state files are not evidence of live processes.
Original run records verified why the Qwen0.8 queue stopped: the controllers exited during idle-GPU checks before launching the next shards. Original run evidence is retained locally.
Snapshot fingerprint
SHA256 of the original complete aggregate JSON:
All displayed data is embedded in this HTML, which can be saved, shared, and opened offline on its own. The public page consists only of this file; original run evidence is retained locally. Scores and coverage retain the October 13, 2026 snapshot. FullKV output lengths were recovered from retained run records for those snapshot IDs; the length protocol states the extraction time.