Quantisation
quantization · INT8 · INT4
กล่าวโดยย่อ
Quantisation reduces the numerical precision of a model's weights — for example from 16-bit floating point to 8-bit or 4-bit integers — so the model occupies less memory and runs faster. Quality loss is small at 8-bit and becomes noticeable, though often acceptable, at 4-bit.
A model's weights are stored as numbers. Training typically uses 16-bit floating point. Quantisation re-encodes them at lower precision — 8-bit or 4-bit integers are common — with scaling factors that map the reduced range back onto the original one.
Two benefits follow, and the second is the bigger one. Memory shrinks roughly in proportion, so a model that needed several accelerators may fit on one. And because decoding is bound by memory bandwidth rather than arithmetic, moving fewer bytes per token directly increases generation speed.
Methods differ in where the work happens. Post-training quantisation converts an already-trained model, sometimes using a small calibration dataset to choose scaling factors. Quantisation-aware training simulates the reduced precision during training so the model adapts to it, giving better results at very low bit widths for considerably more effort.
The practical caution is that published quality claims are averages over general benchmarks. Degradation is not uniform: long-context and multi-step reasoning tasks tend to suffer first. If quantisation is part of your cost plan, measure it on your own evaluation set before committing.
คำถามที่พบบ่อย
- How much quality is lost?
- At 8-bit, usually little enough to be hard to detect on ordinary tasks. At 4-bit, degradation is measurable and shows up first on reasoning-heavy and long-context work. Always evaluate on your own tasks rather than trusting a general claim.
- Why does quantisation make inference faster?
- Token generation is limited by memory bandwidth, not arithmetic. Smaller weights mean less data moved per token, so the model produces output faster even though the number of operations is unchanged.