Skip to content

Handle F32 x F16 src1 in CUDA binary broadcast ops - #1591

Open
Bonenk wants to merge 1 commit into
ggml-org:masterfrom
Bonenk:fix/binbcast-f32-f16-broadcast
Open

Handle F32 x F16 src1 in CUDA binary broadcast ops#1591
Bonenk wants to merge 1 commit into
ggml-org:masterfrom
Bonenk:fix/binbcast-f32-f16-broadcast

Conversation

@Bonenk

@Bonenk Bonenk commented Aug 20, 2026

Copy link
Copy Markdown

Problem

ggml_cuda_op_bin_bcast and ggml_cuda_op_fused_binbcast_impl dispatch on the
types of src0 and dst only. The first branch matches any F32 src0 / F32 dst
pair and instantiates src1_t = floatwithout verifying src1's type.

When src1 is actually F16 (for example an F16 per-channel scale or
LayerScale weight broadcast over F32 activations), the kernel reads the tensor
with float strides. launch_bin_bcast_pack detects the mismatch via the
nb10 % sizeof(src1_t) == 0 alignment assert and aborts:

binbcast.cu:293: GGML_ASSERT(nb10 % sizeof(src1_t) == 0) failed

The F16 src0 / F32 src1 mixed case has always been handled; only the
F32 src0 / F16 src1 combination was missing. The same pattern exists in both
the regular and the fused dispatcher.

Change

  • Require src1 == F32 in the pure F32/F32 branch (it previously matched any
    src1).
  • Add the missing F32 src0 / F16 src1 / F32 dst branch to both the regular
    and the fused dispatchers, reading src1 as half so the F16 tensor stays
    in place (no upcast copy needed).

Testing

  • Reproduced the assert with an F32×F16 broadcast graph on an Ampere GPU
    (sm_86, CUDA 12.6); the graph now computes correctly.
  • Existing mixed-type paths (F32/F32, F16/F16, F16/F32) are unchanged —
    the fix only adds a branch and tightens an existing condition.

Notes

  • The patch was validated against the ggml version pinned by the
    CrispStrobe/ggml fork (4d9a5c3d, v0.10.2-462). The same bug exists in
    upstream ggml-org/ggml master, so the fix should apply cleanly to both.

Dispatch in ggml_cuda_op_bin_bcast and ggml_cuda_op_fused_binbcast_impl
depended only on src0/dst types, so the F32/F32 branch matched any src1
and instantiated src1_t = float. An F16 src1 then tripped the
nb10 % sizeof(src1_t) alignment assert in launch_bin_bcast_pack.

Require src1 == F32 in that branch and add the missing F32 src0 / F16
src1 / F32 dst case, reading src1 as half in place.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant