Revert "[Bugfix][Gemma] Conditionally create KV projections/norms on KV-shared layers" (#54917) - #55789
Draft
vllm-agent wants to merge 1 commit into
Draft
Revert "[Bugfix][Gemma] Conditionally create KV projections/norms on KV-shared layers" (#54917)#55789vllm-agent wants to merge 1 commit into
vllm-agent wants to merge 1 commit into
Conversation
…KV-shared layers (vllm-project#54917)" This reverts commit f2d45f2.
|
I can reproduce this on an RTX 3090 with Gemma-4 E2B using mixed FA2/FlashInfer backends. I’m investigating a targeted fix and regression tests. Please let me know if someone is already working on it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reverts #54917 (
f2d45f26).Why
:nvidia: (B200) Spec Decode Speculators + MTP Nightlywent red on nightly build #87584 (537af2c3). It was green on the two previous runs of this step (87458, 87494).test_mtp_correctness[gemma4-e4b]now dies at engine init:#54917 changed the KV-shared branch of both
gemma4.pyandgemma3n.pytobut neither
vllm/v1/attention/backends/flashinfer.pynorvllm/v1/attention/backends/flash_attn.pyguards aNonekey/value — both slicekey[:num_actual_tokens]/value[:num_actual_tokens]unconditionally. So every Gemma 4 E2B/E4B and Gemma 3n model withnum_kv_shared_layers > 0now crashes at engine init on the two main attention backends. The PR's tests are CPU-only weight-loading unit tests, so nothing in the PR exercised the new forward path.f2d45f26is the only commit touchinggemma4.pyin theed29dfae..537af2c3range, and it is the commit that introduces the literalself.attn(q, None, None)line.A narrow fix is preferable to this revert
#54917 fixes a real problem (fine-tuned /
save_pretrainedcheckpoints omitk_proj/v_proj/k_normon KV-shared layers). Please land a forward fix instead of merging this if you can — either option keeps that fix:key is None/value is Nonewhenkv_sharing_target_layer_nameis set (skip the slicing and thereshape_and_cachewrite), orforwardpassing realk/vtensors through toself.attnwhile still not creating thek_proj/v_proj/k_normmodules.This draft revert is only a fallback to unblock the nightly if no forward fix lands.
Details
537af2c3a4ba7462ddc9bc94ec7a4ea496da6d2e):nvidia: (B200) Spec Decode Speculators + MTP NightlyAuto-generated by CI failure analyzer.