brain

cafe-llama.cpp

Properties14
descriptionA experimental fork of llama.cpp with LLM inference in C/C++
statusseedling
typesource
created2026-09-28
updated2026-09-28
tagsoss
aliases
sourcehttps://github.com/quimmedes/cafe-llama.cpp
homepagehttps://llama.app
docs_urlhttps://github.com/quimmedes/cafe-llama.cpp/wiki
languageC++
licenseMIT
stars49
public

14 min read

llama.cpp

ilustration

MoE Offloading & Memory Optimization

In Mixture of Experts (MoE) models (such as Qwen 3.8 Flash Next, DeepSeek-V2/V3, Mixtral, etc.), expert weights represent the majority of parameters and VRAM. cafe-llama.cpp provides flags to offload MoE weights to pinned host RAM (CUDA_Host) or CPU RAM while keeping attention, KV cache, and routers on the GPU:

FlagLong FlagDescription
--pipeline-parallel--no-pipeline-parallelEnable the offloading acceleration pipeline, required for -hmoe, -nhmoe, -hmoed.
-hmoe--host-moeKeep all MoE expert weights in pinned host memory (CUDA_Host). Enables zero-copy async DMA over PCIe.
-nhmoe N--n-host-moe NKeep MoE weights of the first N layers in pinned host memory.
-cmoe--cpu-moeKeep all MoE expert weights in CPU system RAM.
-ncmoe N--n-cpu-moe NKeep MoE weights of the first N layers in CPU system RAM.
-ssd--ssd-streaming, --no-ssd-streamingStream routed expert weights from SSD on-demand via mmap: only the experts a token actually selects page into RAM, the rest stay on disk.
-nssd N--ssd-n-streaming NStream the MoE experts of the first N layers from SSD (implies --ssd-streaming); analogous to -ncmoe / -nhmoe, but the destination is SSD instead of RAM.
-hmoed--host-moe-draftKeep draft model MoE weights in pinned host memory (for speculative decoding).
-nhmoed N--n-host-moe-draft NKeep draft model MoE weights of the first N layers in pinned host memory (for speculative decoding).
-cmoed--cpu-moe-draftKeep draft model MoE weights in CPU system RAM (for speculative decoding).
-ncmoed N--n-cpu-moe-draft NKeep draft model MoE weights of the first N layers in CPU system RAM (for speculative decoding).

Turbo KV Cache (low-bit K/V, GPU-only)

Low-bit TurboQuant types for the attention KV cache, selected with -ctk / -ctv (draft: -ctkd / -ctvd). Rows are stored in the rotated (WHT) domain and reconstructed inside the fused flash-attention kernel, so flash-attention (-fa) is required: it is auto-enabled, and loading aborts if flash-attention is explicitly disabled. Turbo KV types are GPU-side only.

TypeApprox bits/valueCompression vs fp16Constraint
turbo4~4.1~3.9xnone
turbo3~3.5~4.6xhead dim (K/V) must be a multiple of 128
turbo2~2.5~6.4xhead dim (K/V) must be a multiple of 128

Lower-bit K/V lets long-context KV fit in VRAM. Example: -ctk turbo4 -ctv turbo4 -fa on.

Qwen 3.8 Flash Next / Qwen4 Internal N-Gram (PLE) Optimization

Qwen 3.8 Flash Next includes an internal Prompt Lookup Expert (PLE) N-gram hash embedding table (per_layer_token_embd, ~51B parameters). cafe-llama.cpp provides dedicated flags to manage its memory footprint:

FlagLong FlagDescription
--ngram-ssd--offload-ngram-ssd, --no-ngram-ssdExclusively offload the internal N-gram embedding table to SSD on-demand via memory mapping (mmap), leaving active model layers in RAM/VRAM.
--no-ngram--disable-ngram, --no-load-ngramForce completely disable the internal N-gram embedding table and PLE layers, skipping all PLE tensors (0 bytes allocated in RAM/VRAM).
--ngram--load-ngramNormal mode: load internal N-gram table and PLE layers into memory (default).

Qwen 3.8 Flash Next & MTP Speculative Decoding

MTP (Multi-Token Prediction) draft models in GGUF format are available at: 👉 Hugging Face: quimmedes/Qwen3.8-Flash-Next-MTP-GGUF

Available quantizations:

  • mtp-Qwen3.8-Flash-Next-Q4_K_M.gguf (~2.65 GB) - Recommended for balanced memory and speed
  • mtp-Qwen3.8-Flash-Next-Q6_K.gguf (~3.24 GB)
  • mtp-Qwen3.8-Flash-Next-Q8_0.gguf (~3.94 GB)
  • mtp-Qwen3.8-Flash-Next-BF16.gguf (~7.40 GB) - Full precision

Recommended Server Command for Qwen 3.8 Flash Next:

llama-server -m Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf \
-ctk q8_0 -ctv q8_0 -kvu \
 -fa on -ngl 99 -nhmoe 36 -c 64000 --pipeline-parallel -np 1 --no-ngram 

Disable Ngram if you don't have enough RAM/VRAM
llama-server -m Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf \
-ctk q8_0 -ctv q8_0 -kvu \
 -fa on -ngl 99 -nhmoe 36 -c 64000 \
--no-ngram --pipeline-parallel -np 1 --no-ngram 

MTP with offload
llama-server \
  -m Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf \
  -md mtp.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  -ngl 99 \
  -nhmoe 36 \
  -fa on \
  -ctk q8_0 -ctv q8_0 -kvu \
  -ctkd q4_0 -ctvd q4_0 -ngld 99 \
  -c 64000 -b 1024 -ub 128 -np 1 \
  --pipeline-parallel --no-ngram 
  
  
  //Faster above 30% context load
  
  llama-server -m Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf \
  -md mtp.gguf -ctk q8_0 -ctv q8_0 -kvu -fa on -c 64000 \
  -np 1 -t 8 -b 1024 -ub 128 --spec-type draft-mtp,ngram-mod \
  --spec-draft-n-max 2 --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48 \
  --spec-ngram-mod-n-max 64 -ctkd q4_0 -ctvd q4_0 \
  --pipeline-parallel -nhmoe 34 -ngl 99 -ngld 99 --no-ngram

Serving Safetensors Checkpoints

-m / --model accepts a Hugging Face safetensors checkpoint directly: either a directory with config.json and its shards, or a single .safetensors file. No GGUF file is produced - the checkpoint metadata is read into memory and the weights are streamed from the shards while the model buffers are filled, so the shards stay untouched and no extra disk space is used.

llama-server -m /path/to/Qwen3.6-35B-A3B-FP8 -c 32768
FlagDescription
--safetensors-outtype TYPEStorage type for the quantized weights of the checkpoint: auto (default) keeps FP8 weights in q8_0 and packed int4 weights in q4_0, so neither expands to 16 bits. f16, bf16, q8_0 and q4_0 force one type for every quantized weight. NVFP4 weights are always kept in nvfp4, the flag does not apply to them.
--safetensors-nativeKeep the FP8 weights of the checkpoint as they are stored, in the f8_e4m3 type: the E4M3 codes plus the fp32 scale of every block of 128 values, 8.25 bits per weight. This is the same as --safetensors-outtype native, and it takes precedence over everything else the outtype asks for.

Notes:

  • -mm / --mmproj accepts the same checkpoint: the vision tower is built in memory as a CLIP model, so images work without a converted mmproj-*.gguf. The capabilities of the tower are read from the same in memory metadata, the checkpoint is never opened as a GGUF file.
  • MTP: --spec-type draft-mtp uses the MTP layer that is part of the checkpoint, and -md <checkpoint> loads only its MTP block as a standalone draft (-md with a safetensors path is detected automatically). The block can sit in the main shards or in a shard of its own, like the model_mtp.safetensors of Qwen3.5.
  • Weights stored as-is in the checkpoint (BF16, F16, F32) are read with mmap and take no extra RAM, exactly like a GGUF file. Everything else is produced by the source while the model buffers are filled: FP8 is dequantized with its 128x128 block scales or its per tensor scale, compressed-tensors int4 weights are unpacked with their per group scales, NVFP4 weights are repacked in the nvfp4 blocks with their E4M3 scales, packed experts are collected into a single 3d tensor, norms and linear attention parameters are adjusted, and the vision tower is cast to F16/F32 - the same work the converter does, in memory. For a 35B FP8 checkpoint that is about 35 GB of RAM with q8_0 (nothing is written to disk).
  • --safetensors-native keeps FP8 weights in f8_e4m3 instead of requantizing them to q8_0, so the codes and the block scales of the checkpoint survive bit for bit. The CPU vector kernels decode the codes directly, CUDA dequantizes them in the kernel and Vulkan in the shader. The type has no matrix-matrix kernels: on both GPU backends batches of up to 8 columns use the vector kernels, larger ones the dequantize to_f16 path. CUDA offloads routed experts (MUL_MAT_ID) of MoE checkpoints as well, Vulkan does not, so on Vulkan the experts of a MoE checkpoint stay on the CPU. SYCL, Metal and WebGPU do not implement the type.
  • The block of f8_e4m3 covers 128 values and holds one fp32 scale, so the row length of every FP8 weight of the checkpoint must be a multiple of 128. The loader stops with an error naming the tensor otherwise.
  • Exl3 checkpoints (ExLlamaV3) are decoded in memory as well: every quantized linear holds a trellis of 1 to 3 bit coded indices plus one factor per input and per output channel. The loader runs the reference decode (the 16 bit sliding window through the mul1 codebook, then the Hadamard transform of 128 along both axes with the factors) and emits a dense weight. The output type follows the bit width of each tensor - 1 and 2 bit tensors become q2_k, 3 bit tensors q3_k - so a 2.0 bpw 27B checkpoint lands near its own file size (about 12 GB resident for a 9.7 GB checkpoint). --safetensors-outtype q8_0, f16, bf16 or q4_0 overrides that with one higher precision type for every tensor when memory is not the constraint.
  • The native path is verified against the same checkpoint loaded on the CPU: CUDA reproduces the CPU logits (Mean KLD -0.000000, 100 % identical top tokens), Vulkan stays within its own rounding (0.000108, 99.2 % identical top tokens).
  • A modelopt MIXED_PRECISION checkpoint quantizes the attention layers to FP8 and the FFN and the LM head to NVFP4, layer by layer. The NVFP4 weights keep their original bits and their weight_scale_2 and input_scale are loaded as the separate <name>.scale and <name>.input_scale tensors that the graph multiplies with, exactly like the converter stores them.
  • The tokenizer is read from vocab.json and merges.txt when the checkpoint ships them, otherwise from tokenizer.json, so checkpoints that only carry the fast tokenizer load as well.
  • The SSD streaming flags (-ssd, -nssd, --ngram-ssd) cannot page those tensors from disk, the loader logs a warning when they are requested.
  • Supported checkpoints with a hand written mapping: qwen3_5 (Qwen3.5 / Qwen3.8 dense), qwen3_5_moe (Qwen3.5 / Qwen3.6 MoE), gemma4 / gemma4_unified (Gemma 4, text model of the unified multimodal checkpoints), NemotronH (Nemotron-H and Nemotron-3 Nano Omni, text model) and agnes (Agnes 3.0), text model and vision tower of the qwen and agnes families.
  • The nemotron-h mapping reads the hybrid layout of that family: the layer pattern (M mamba2, E experts, * attention) decides the per layer feed forward length and kv head count, the mamba2 parameters keep the shapes this runtime expects (the group norm split by the ssm groups, A_log negated and exponentiated, D and A_log as columns), and the routed experts are NVFP4 packed per expert, stacked into one tensor with the per expert scales next to it. Its scale factors are the ones the converter writes: context_length of the hybrid window, the uniform head width for the rope, and the pixtral pre-tokenizer of the checkpoint.
  • The gemma 4 mapping covers what that family needs on top of the plain decoder: the sliding window pattern per layer, the separate head dimensions and rope of the full attention layers, the shared kv layers, the per layer output scale, the key used as value where the checkpoint has no v_proj, and the frequency factors of the proportional rope, which the loader generates like the converter does. Its tokenizer is read as the sentencepiece style pieces of this repo (gemma4 tokenizer), the token ids match the ones of the Hugging Face tokenizer.
  • A checkpoint whose tensor names do not line up with what the architecture expects is refused with the name of the missing tensor, and a checkpoint that mixes several compressed-tensors groups (e.g. FP8 attention with NVFP4 FFN, as the unsloth NVFP4 releases do) is handled per tensor: the group of every tensor is resolved by name, and the global scales of the NVFP4 groups are applied as the reciprocal ggml expects.
  • Any other checkpoint of a plain decoder is read by a generic mapping: the tensor names, the hyperparameters and the tokenizer come from config.json and the tokenizer files. The families known to it are llama, mistral and mixtral (llama), qwen2 and qwen2-moe, qwen3 and qwen3-moe, gemma, gemma2 and gemma3-text, olmo2, phi3 and starcoder2. It applies what those layouts need on top of the plain skeleton: the (1 + w) normalization of the gemma family, the q/k rotary permutation of the llama family, the attention biases of qwen2 and the routed experts of the MoE variants. This is pure data derived from the configuration, no per architecture code, and it is verified against the converter: loading such a checkpoint and the GGUF produced from it gives the same perplexity and logits.
  • A checkpoint that needs more than that - fused or reshaped projections, a tokenizer that is not byte pair encoding (SentencePiece, WordPiece, tiktoken), or a non decoder layout - is refused with the reason instead of being loaded wrongly. Convert those to GGUF first.
  • Packed int4 checkpoints (compressed-tensors pack-quantized, 4 bit weights, one scale per group of 128 columns) are loaded without expanding them to 8 or 16 bits. The routed experts of a packed checkpoint are not supported yet.
  • NVFP4 weights are repacked from the modelopt nibble order into the ggml nvfp4 blocks, which need a row length that is a multiple of 64. Packed NVFP4 experts are not supported yet.
  • Agnes 3.0 runs a second FFN in parallel with the main one. It is loaded as separate tensors (ffn_gate_par, ffn_up_par, ffn_down_par) with the length in the new *.feed_forward_parallel_length key, and the graph adds its output to the main FFN - so every weight can be mapped from the checkpoint instead of being rewritten.
  • Without -nr the fork copies the weights of CPU layers into repacked host buffers; add -nr to keep them memory mapped when serving a checkpoint that does not fit in RAM, and -ngl N to place part of the layers in VRAM.

Building from Source

1. NVIDIA CUDA (Windows / Linux)

# CMake configure with CUDA backend
cmake -B build -DGGML_CUDA=ON

# Build Release
cmake --build build --config Release -j 2

2. Vulkan (Cross-Platform AMD / Intel / NVIDIA)

# Requires Vulkan SDK installed
cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release -j 2

3. AMD ROCm / HIP (Linux / Windows)

cmake -B build -DGGML_HIP=ON -DAMDGPU_TARGETS="gfx1100;gfx1030"
cmake --build build --config Release -j 2

4. Apple Metal (macOS)

cmake -B build -DGGML_METAL=ON
cmake --build build --config Release -j 2

5. CPU Only (AVX2 / AVX-512)

cmake -B build -DGGML_CUDA=OFF -DGGML_VULKAN=OFF
cmake --build build --config Release -j 2

-j N: The number of CPU threads the processor will use to compile, a safe number is the amount of physical cores of the processor.

Quick start

A few options to get llama.cpp installed on your machine:

Once installed:

# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
VLM session with llama cli VLM session with llama cliBuilt-in web UI against llama serve running Qwen 3.6 Built-in web UI against llama serve

Description

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

  • Plain C/C++ implementation without any dependencies
  • Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
  • AVX, AVX2, AVX512 and AMX support for x86 architectures
  • RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
  • 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
  • Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
  • Vulkan and SYCL backend support
  • CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is build on top of the ggml library.

Supported backends

BackendTarget devices
BLASAll
BLISAll
CANNAscend NPU
CUDANvidia GPU
HIPAMD GPU
HexagonSnapdragon
IBM zDNNIBM Z & LinuxONE
MUSAMoore Threads GPU
MetalApple Silicon
OpenCLAdreno GPU
OpenVINO [In Progress]Intel CPUs, GPUs, and NPUs
RPCAll
SYCLIntel GPU
VirtGPUVirtGPU APIR
VulkanGPU
WebGPUAll
ZenDNNAMD CPU

Documentation

Tools

Development

Contributing

  • Contributors can open PRs
  • Collaborators will be invited based on contributions
  • Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
  • Any help with managing issues, PRs and projects is very appreciated!
  • Read the CONTRIBUTING.md for more information

Acknowledgements

  • yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
  • nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
  • nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
  • mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
  • sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain

Summary

MoE-offloading llama.cpp fork (49 stars, MIT, quimmedes): flags to keep MoE expert weights in pinned host RAM (-hmoe, zero-copy PCIe DMA), CPU RAM (-cmoe), or SSD-streamed via mmap (-ssd — only token-selected experts page in), with first-N-layers variants and -hmoed drafter offload; attention/KV/routers stay on GPU. Targets Qwen3.8-Flash-Next, DeepSeek.

Learnings

  • Strata’s tiered-offload idea as llama.cpp flags on any MoE GGUF instead of a bespoke engine — the community answer for running 125B-class MoE on 3090+32GB-class rigs. In Inference engines’s forks list; pairs with the Qwen3.8-Flash-Next section of Qwen model family.

Gotchas

  • (none yet)

Hands-on

(candidate for llamacpp-tools build script; the 125B-on-doombox experiment fork)

Graph View

Backlinks