GSQ-RCO
GSQ-RCO is a quantization method from IST Austria’s DASLab. Two halves: GSQ (Gumbel-Softmax Quantization) learns optimal low-bit grids and scales per weight; RCO (Riemannian Constrained Optimization) allocates bits non-uniformly across tensors under an exact size budget. Together they produce high-quality standard GGUF models that run in stock llama.cpp — often near task-lossless. (User note, 2026-10-07.)
The two halves
- GSQ — learned grids, not fixed ones. Classic quants (llama.cpp k-quants, exl) place weight values on hand-designed grids (symmetric/asymmetric levels, fixed scales). GSQ instead treats grid points and scales as trainable: straight-through Gumbel-Softmax relaxation lets gradient descent discover the value placements a tensor actually needs — non-uniform, per-weight-group, data-aware.
- RCO — the bit budget as a constraint, not a guess. Mixed-precision question: which tensors get 4 bits, which get 2? RCO optimizes this allocation on a Riemannian manifold (the geometry of the constraint surface) so the total lands exactly on the size budget — no post-hoc rounding that overshoots or wastes space. Sensitive layers keep precision; robust ones give it up.
Why it matters (the local-inference lane)
- Stock-llama.cpp compatibility: output is ordinary GGUF — no forked runtime, no custom kernels (contrast: bitshift-trellis 2.4-bit in Inference engines needs patched ggml types; PTQ1_0 ternary needs a PrismML llama.cpp fork). GSQ-RCO’s quality-per-bit arrives everywhere llama.cpp already runs — including llama-swap slots.
- The reference artifacts: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF — the quant Strata bundles as its default; UkisAI/Swift-1.5 fine-tune rides the same quant (Qwen model family, Strata (OSS), both captured).
- Honest limits, from the vault’s own capture: at the extreme low end (~2 bits), learned grids alone don’t save you — the llama.cpp-mirai-s comparison ([Inference engines], captured) has GSQ-RCO IQ2_XS collapsing to 33.3% AppWorld where codebook-at-runtime 2.4-bit scores 57.9% and beats bf16. GSQ-RCO’s sweet spot is the 3–5 bit / near-lossless band; the sub-3-bit frontier needs runtime-computed codebooks.
Position in the quant landscape
| Method | Grid | Bits | Runtime | Kin |
|---|---|---|---|---|
| k-quants / i-quants (llama.cpp) | fixed, hand-designed | 2–8 | stock llama.cpp | the incumbent |
| GSQ-RCO | learned per-weight | ~2–5 (sweet 3–5) | stock llama.cpp | this note |
| GPTQ/AWQ | learned scales, fixed structure | 2–4 | engine-specific | the GPU-lane classics |
| Bitshift-trellis (Mirai) | runtime-computed codebook | 2–3 | patched ggml | Inference engines |
| PTQ1_0 ternary (PrismML) | {-1,0,1} + serve surgery | 1.58 | PrismML fork | Local LLM inference |
Vault kin
- Local LLM inference — the homelab lane where GSQ-RCO quants actually run (llama-swap)
- Inference engines — the 2.4-bit frontier comparison + Strata bundling
- Qwen model family — Swift 1.5 / Qwen3.8-Flash-Next quants in active use
- Strata (OSS) — ships GSQ-RCO as the default compressed model
- Embedding models — quantization pressure is lower there, but the learned-grid idea transfers