Handle F32 x F16 src1 in CUDA binary broadcast ops - #1591
Open
Bonenk wants to merge 1 commit into
Open
Conversation
Dispatch in ggml_cuda_op_bin_bcast and ggml_cuda_op_fused_binbcast_impl depended only on src0/dst types, so the F32/F32 branch matched any src1 and instantiated src1_t = float. An F16 src1 then tripped the nb10 % sizeof(src1_t) alignment assert in launch_bin_bcast_pack. Require src1 == F32 in that branch and add the missing F32 src0 / F16 src1 / F32 dst case, reading src1 as half in place.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ggml_cuda_op_bin_bcastandggml_cuda_op_fused_binbcast_impldispatch on thetypes of
src0anddstonly. The first branch matches anyF32 src0 / F32 dstpair and instantiates
src1_t = float— without verifyingsrc1's type.When
src1is actuallyF16(for example an F16 per-channel scale orLayerScale weight broadcast over F32 activations), the kernel reads the tensor
with float strides.
launch_bin_bcast_packdetects the mismatch via thenb10 % sizeof(src1_t) == 0alignment assert and aborts:The
F16 src0 / F32 src1mixed case has always been handled; only theF32 src0 / F16 src1combination was missing. The same pattern exists in boththe regular and the fused dispatcher.
Change
src1 == F32in the pure F32/F32 branch (it previously matched anysrc1).F32 src0 / F16 src1 / F32 dstbranch to both the regularand the fused dispatchers, reading
src1ashalfso the F16 tensor staysin place (no upcast copy needed).
Testing
(sm_86, CUDA 12.6); the graph now computes correctly.
the fix only adds a branch and tightens an existing condition.
Notes
CrispStrobe/ggmlfork (4d9a5c3d, v0.10.2-462). The same bug exists inupstream
ggml-org/ggmlmaster, so the fix should apply cleanly to both.