Skip to content

Revert "[Bugfix][Gemma] Conditionally create KV projections/norms on KV-shared layers" (#54917) - #55789

Draft
vllm-agent wants to merge 1 commit into
vllm-project:mainfrom
vllm-agent:auto-revert/pr-54917
Draft

Revert "[Bugfix][Gemma] Conditionally create KV projections/norms on KV-shared layers" (#54917)#55789
vllm-agent wants to merge 1 commit into
vllm-project:mainfrom
vllm-agent:auto-revert/pr-54917

Conversation

@vllm-agent

Copy link
Copy Markdown
Contributor

Reverts #54917 (f2d45f26).

Why

:nvidia: (B200) Spec Decode Speculators + MTP Nightly went red on nightly build #87584 (537af2c3). It was green on the two previous runs of this step (87458, 87494). test_mtp_correctness[gemma4-e4b] now dies at engine init:

File "vllm/model_executor/models/gemma4.py", line 1308, in forward
File "vllm/model_executor/layers/attention/attention.py", line 767, in unified_attention_with_output
File "vllm/v1/attention/backends/flashinfer.py", line 2047, in forward
    key = key[:num_actual_tokens]
TypeError: 'NoneType' object is not subscriptable

#54917 changed the KV-shared branch of both gemma4.py and gemma3n.py to

attn_output = self.attn(q, None, None)

but neither vllm/v1/attention/backends/flashinfer.py nor vllm/v1/attention/backends/flash_attn.py guards a None key/value — both slice key[:num_actual_tokens] / value[:num_actual_tokens] unconditionally. So every Gemma 4 E2B/E4B and Gemma 3n model with num_kv_shared_layers > 0 now crashes at engine init on the two main attention backends. The PR's tests are CPU-only weight-loading unit tests, so nothing in the PR exercised the new forward path.

f2d45f26 is the only commit touching gemma4.py in the ed29dfae..537af2c3 range, and it is the commit that introduces the literal self.attn(q, None, None) line.

A narrow fix is preferable to this revert

#54917 fixes a real problem (fine-tuned / save_pretrained checkpoints omit k_proj/v_proj/k_norm on KV-shared layers). Please land a forward fix instead of merging this if you can — either option keeps that fix:

  1. Make the attention backends accept key is None / value is None when kv_sharing_target_layer_name is set (skip the slicing and the reshape_and_cache write), or
  2. Keep the shared-layer forward passing real k/v tensors through to self.attn while still not creating the k_proj/v_proj/k_norm modules.

This draft revert is only a fallback to unblock the nightly if no forward fix lands.

Details

  • Build: #87584 (537af2c3a4ba7462ddc9bc94ec7a4ea496da6d2e)
  • Failing job: :nvidia: (B200) Spec Decode Speculators + MTP Nightly
  • New failures linked to this PR: 1
  • Reverts cleanly: 3 files, +99/-222 (exact inverse of the original).

Auto-generated by CI failure analyzer.

@mergify mergify Bot added the bug Something isn't working label Sep 8, 2026
@lzzzzzc

lzzzzzc commented Sep 8, 2026

Copy link
Copy Markdown

I can reproduce this on an RTX 3090 with Gemma-4 E2B using mixed FA2/FlashInfer backends. I’m investigating a targeted fix and regression tests. Please let me know if someone is already working on it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants