FVQ · Semantically Lossless Tokens

§2.1 · Finding 1

Does lower error mean better understanding?

Higher reconstruction error, yet better downstream understanding.

Finding 1Reconstruction error alone is not a reliable measure of semantic fidelity.

Loading comparison…

§2.2 · Finding 2

Bits are the budget for meaning

Raw pixels, continuous features and discrete indices are all finite-bit codes. A discrete token spends B = L · log₂C bits: linear in the number of codebooks L, only logarithmic in codebook size C. So the cheap way to grow the budget is to add codebooks, and understanding climbs as it grows. Until it stops: RVQ and PQ flatten between 112 and 224 bits, because each added codebook gets less to work with.

Finding 2Larger bit budgets raise semantic capacity when the added codes stay informative, far enough for discrete tokens to match continuous ones.

Average of 8 benchmarks

VQRVQPQFVQContinuous

Reading examples 14-bit examples

Fig. 1(b) · hover to zoom

Answers at 14 bits.

§3 · Method

Factorized Vector Quantization

A quantizer that keeps scaling needs three things. Every codebook plays a symmetric role. Every codebook gets a fixed-dimensional view of the full feature, so adding codebooks never thins out what each one sees. And the views together preserve the feature exactly before quantization. FVQ gets all three from a tight frame: fixed dense projections, parallel nearest-neighbour lookups, and a synthesis operator that inverts the projections.

What each added codebook gets to see

L =

Schematic of the three designs described in §3, not measured values.

Factorized Vector Quantization

The paper’s method figure · animated walkthrough

The complete FVQ architecture

Original figure from the paper. Replay to reveal its components, then follow each highlighted stage.

Analysis · tight frame

u_i,ℓ = R_ℓz_i, A^⊤A = ρI_D, ρ = Ld⁄D

L dense projections Rℓ ∈ ℝ^d×D stacked into A. The tight-frame condition scales every direction equally, so no direction is lost by all views at once. Built once by QR of a fixed Gaussian matrix and never trained.

Symmetric parallel lookup

k_i,ℓ = arg min_j ‖u_i,ℓ − c_ℓ,j‖²

Each view is matched against its own codebook, independently and in parallel. A patch becomes the tuple (k₁,…,k_L) worth B = L log₂C bits, with no residual hierarchy.

Synthesis · exact before quantization

ẑ_i = 1⁄ρ Σ_ℓ R_ℓ^⊤c_{ℓ,k_i,ℓ}

Since ρ⁻¹AᵀA z = z, the factorization adds no error of its own. All distortion comes from snapping views to codewords. After tokenizer training, only the indices go to the VLM, through new embeddings learned from scratch.

§4.1 · Recipe

Four stages, 200B tokens

The tokenizer learns to reconstruct frozen Qwen3 ViT features, with no pixel loss. It then stays frozen while the language side learns to read its indices. Bar widths below are proportional to training tokens.

stage 1

Quantizer training

100B tokens

VFMprojectioncodebooksfeature decodertight frame

Reconstruct frozen Qwen3-VL-8B ViT features. Tight-frame projections stay fixed.

stage 2

Alignment

10B tokens

tokenizerLLMvisual embeddingsmerger

Map discrete IDs into the language embedding space.

stage 3

Pre-training

60B tokens

tokenizerLLMvisual embeddingsmerger

8K context, images up to 1024 px.

stage 4

Supervised fine-tuning

30B tokens

tokenizerLLMvisual embeddingsmerger

16K context, images up to 2048 px.

LLM Qwen3-8BVision Qwen3 ViT + 16-codebook FVQ, any resolutionPositions standard 1-D RoPERemoved DeepStack connectionsOptimizer AdamW, β = (0.9, 0.96)trainedfrozen

§4.2 · Main results

A discrete model in continuous company

Nine-metric average over MMMU, MathVista, MMStar, OCRBench/10, ChartQA, CharXiv-DQ, CharXiv-RQ, DocVQA and InfoVQA. Hatched bars are discrete-token models. FVQ ties Qwen3-VL-8B on MathVista, posts the best RealWorldQA score in the table, and beats every continuous model listed on both CharXiv tasks.

Nine-metric average

STEM & general understanding

* evaluated by the authors under the same setting. OmniDocBench is an edit distance, lower is better. Amber marks the best discrete result.

§4.3 · Fine-grained cases

Reading at the limit of resolution

Explore nine examples from the paper and supplementary material. Follow the highlighted image regions, then inspect the complete recorded model responses.

Loading interactive cases…

§4.4 · Ablations

The bit-budget effect is not a QwenViT quirk

Moving from one VQ codebook (14 bits) to eight FVQ codebooks (112 bits) lifts every encoder tested. Without any industrial-scale data, FVQ trained only on LLaVA-OneVision SFT still beats ViQ, the strongest discrete baseline in that setting.

14 → 112 bits across encoders

Tables 2 & 5 · 8-bench avg

LLaVA-OneVision SFT only

Table 6 · Ours vs ViQ

Same Qwen2.5-7B backbone family, no industrial pre-training. Scores for Ours, with the gain over ViQ.

Cite

BibTeX

@misc{zou2026discrete,
  title         = {Can Discrete Tokens Serve as Semantically Lossless
                   Representations of Visual Signals?},
  author        = {Zou, Jialv and Li, Xiaotong and Yue, Zhengrong and
                   Huang, Xu and Luo, Yaxin and Yan, Wenhao and Li, Haodong
                   and Liu, Wenyu and Wang, Xinggang},
  year          = {2026},
  eprint        = {2610.37662},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2610.37662}
}