🛠️[v0.4.1] Release Note: MTMD Template Compatibility, Vision/Video Input Improvements, and Expanded GGML Backend Bindings

[JamePeng]

[0.4.1] Release Note: MTMD Template Compatibility, Vision/Video Input Improvements, and Expanded GGML Backend Bindings

This release focuses on MTMD compatibility, runtime reliability, and backend coverage.

Generic MTMD handling is expanded for more vision templates and media input formats, including Muse Glimmer, Qwen3-VL, and Qwen3.5 video inputs. It also fixes model-template resolution, standard Jinja helpers, media evaluation diagnostics, and the MTMD decoder-position ABI layout.

On the backend side, ggml-backend.h bindings are significantly expanded to cover device, buffer, tensor, graph, event, and scheduler APIs, while the README and Wiki have been updated to reflect the current bundled llama.cpp behavior and build options.

• feat(mtmd): expand vision chat template media support

  • Recognize Muse Glimmer's <|patch|> marker for image detection in the generic MTMD handler
  • Update Qwen3VLChatHandler rendering to inject image and video URLs for MTMD processing.
  • Enable Qwen35ChatHandler video inputs and handle string or {"url": ...} values.
  • Add regression tests for the supported media input formats.

• feat(ggml): expand backend API bindings

  • Bind device, buffer, tensor, graph, event, and scheduler APIs from ggml-backend.h.
  • Correct the device initializer and buffer base pointer signatures, and document each binding with its C declaration and usage notes.

• fix(mtmd): load and compile the model template for generic chat handlers

  • Preserve unresolved template selection until the first request instead of
    treating the precompiled fallback as an explicit template.
  • Recompile the resolved template before extracting media tags so rendering, placeholder
    normalization, and extra template arguments use the same template.

• fix(mtmd): provide actionable guidance for media evaluation failures

  • Include the native error code, start position, media token count, context size, and batch size.
  • For video requests, suggest reducing frame sampling and image token limits or increasing the context size.
  • Clarify that video options belong on an explicitly constructed chat handler.
  • Add regression tests for video and non-video error messages.

• fix(mtmd): register standard Jinja chat template helpers

  • Expose raise_exception so unsupported inputs return the template's intended error message instead of an undefined-function error.
  • Expose strftime_now for model chat templates that use date formatting.
  • Reuse the existing formatter implementations and add regression tests for both helpers.

• fix(mtmd): align decoder position struct with native ABI

  • Add the missing uint32 z field to restore the 16-byte layout
  • Update type annotations and struct documentation
  • Test field offsets, native struct returns, and array writes

• docs(readme): align feature and build guidance with current behavior

  • Correct CUDA PDL, Metal offload, dynamic backend, and OpenCL build
    guidance against the bundled llama.cpp implementation.
  • Clarify assistant prefill behavior, LoRA request isolation, speculative
    decoding boundaries, and the limits of cross-request state reuse.
  • Update embedding examples to use model-defined pooling and accurately
    describe batching, normalization, output ownership, and generation-state
    cleanup.
  • Use repository-specific server installation commands, document the
    current development test setup, and fix malformed Markdown fences and
    project links.

• docs(wiki): align backend options with bundled llama.cpp

  • Update CUDA documentation to describe PDL as a runtime switch and replace
    obsolete cuBLAS compute variables with GGML_CUDA_CUBLAS_COMPUTE_TYPE.
  • Remove the retired HIP rocWMMA option and correct the current SYCL
    Level Zero and oneDNN variable names.
  • Document dynamic backend dependencies and list the MUSA, KleidiAI,
    WebGPU, and Hexagon build options with their validation boundaries.
  • Clarify that n_gpu_layers=0 disables model-layer offload but does not
    guarantee that all operations avoid an available GPU backend.

Thanks to @KLL535 for the bug report and testing feedback in issue #180.

More information:
5c83af7...3e99f13

-- JamePeng