Skip to content

fix: honor flash attention flag in LLM text encoder attention - #1987

Merged
leejet merged 1 commit into
leejet:masterfrom
linxuhao:fix/llm-flash-attention
Sep 18, 2026
Merged

leejet merged 1 commit into
leejet:masterfrom
linxuhao:fix/llm-flash-attention

Conversation

@linxuhao

Copy link
Copy Markdown
Contributor

Summary

LLMAttention::forward passed a hard-coded false for flash_attn to ggml_ext_attention_ext, so --fa never applied to LLM text encoders (Qwen, Mistral, Gemma, ...), unlike the diffusion models and VAEs that pass ctx->flash_attn_enabled.

Without it the [N*num_heads, n, n] attention matrix is materialized. On Vulkan devices limited to a 4 GiB storage buffer this crosses the limit at ~4096 tokens for 64-head encoders and the ops fall back to CPU (encode 11.5 s at 4065 tokens → 239 s at 4214 tokens on an RX 7900 XTX); on other backends it costs O(n²) memory.

This passes ctx->flash_attn_enabled, matching ernie_image.hpp. Behaviour without --fa is unchanged. The GPT-OSS path (attention sinks) is untouched.

Related Issue / Discussion

Fixes #1986

Additional Information

  • Built sd-cli with -DSD_VULKAN=ON on master cc515a0 with this change.
  • Measured with the same conditioner code on RX 7900 XTX (RADV, Mesa 25.2.8), Qwen3-VL-32B Q4_K_M: 4214 tokens 239 s → 9.7 s, 5081 tokens 357 s → 12.0 s, 15644 tokens 49.7 s; ≤4096-token outputs are identical when --fa is not set.

Checklist

AI-assisted: the investigation, measurements and change were done with Claude Code and reviewed by me.

🤖 Generated with Claude Code

@leejet
leejet merged commit 269e726 into leejet:master Sep 18, 2026
3 of 9 checks passed
@linxuhao
linxuhao deleted the fix/llm-flash-attention branch September 18, 2026 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] LLM text encoders ignore --fa: materialized attention hits Vulkan's 4 GiB buffer limit at 4096 tokens and falls back to CPU

2 participants