Featured image: Contacts on a printed circuit board illustrate computing hardware. Contextual photograph; not an AWQ implementation, quantisation benchmark or a particular AI accelerator. Photo: Vishnu Mohanan / Unsplash. Unsplash licence.
An AI model can become much smaller without removing a single parameter. The trick is to store its numbers with fewer bits. That can make a model fit into available memory, but it also changes the numerical information used to produce its answers.
Quantisation is the process behind that trade-off. This technical explainer uses published AWQ research, SmoothQuant research and current NVIDIA documentation. It does not announce a new October benchmark or a universal performance improvement.
Fewer bits change the storage arithmetic
A bit is a binary digit. Four bits can represent sixteen distinct codes; sixteen bits allow many more. Quantisation uses a numerical mapping, often including scale factors, to represent model values within a smaller set of possibilities.
For an illustrative model with seven billion weights, storing each weight in sixteen bits would require about 14 billion bytes. Four-bit storage would require about 3.5 billion bytes before additional information. This is arithmetic for the weights alone, not a claim that a complete working model needs only 3.5 gigabytes.
The distinction prevents a common misunderstanding. A reduced weight file is one part of deployment. The inference software—the system using the trained model—must still perform the model’s operations and supply the other data needed while it runs.
Weight-only compression makes a specific promise
AWQ, presented at MLSys 2024, addresses low-bit, weight-only quantisation. It uses observations of activations, the intermediate values produced during computation, to identify important weights and select scaling that reduces error.
The researchers found that weights were not equally sensitive to compression. Their approach therefore treats the problem as a search for where numerical detail matters. Its reported performance came from specified models, benchmarks and an accompanying implementation. Those conditions remain part of the result.
A file labelled four-bit does not tell us whether intermediate computations are also four-bit. Weight storage and computation precision can differ. Two deployments carrying the same low-bit label can therefore perform different numerical work.
Unusual values can make compression harder
SmoothQuant, published in the ICML 2023 proceedings, takes a different approach: eight-bit weights and eight-bit activations. Its central technique redistributes the difficulty caused by activation outliers, unusually large values, through a mathematically equivalent transformation before quantisation.
The study reported up to a 1.56-fold speed improvement and a twofold memory reduction in its evaluated settings, with negligible accuracy loss. That is a dated research result under particular conditions. It is not a forecast for every model, processor or request pattern.
The two approaches illustrate why bit width alone is an incomplete comparison. A method has to decide which values to compress, how to map them and how the resulting operations will run efficiently.
Retained accuracy needs an actual evaluation
NVIDIA’s accuracy guidance describes errors that can arise with reduced numerical precision and calculations that are particularly sensitive. It also describes mixed precision, keeping more detail in sensitive operations while using less elsewhere.
Our assessment is that useful deployment comparisons should pair memory and speed measurements with tests representing the intended task. An average benchmark score can be informative while leaving an important local failure mode unexamined. That is an evaluation requirement, not a claim that all quantised models become unreliable.
The comparison should also identify the original model, the compression method, the software and the hardware. Otherwise, an apparent gain cannot be attributed confidently to quantisation itself.
A smaller model must still do the job
Low-bit techniques can make capable models easier to deploy. Their strongest claim is a demonstrated combination: less memory, suitable performance and acceptable retained quality on disclosed tests.
Four-bit storage is a useful engineering tool. It becomes a convincing infrastructure advance when the complete running system preserves the capability that made the larger model worth using.
Leave a Reply