AI glossary
Quantization
Quantization is a model compression technique that reduces the numeric precision used to represent a model’s weights and sometimes its activations, for example converting 32-bit or 16-bit floating-point numbers to 8-bit or 4-bit integers, shrinking the model’s memory footprint and often speeding up inference, usually at some cost to accuracy.
Think of it like compressing a high-resolution image into a smaller file. You lose some fine detail, but the image remains recognizable and loads much faster. In artificial intelligence, this process allows large language models and other neural networks to run on devices with limited memory, such as laptops, smartphones, or even edge devices, without requiring a massive server cluster.
How it works
A model’s weights are normally stored as floating-point numbers with a certain level of precision, such as 32-bit or 16-bit floating point, which determines how much memory each individual number requires. When you train a deep learning model, these weights capture the learned patterns and relationships within the data. Storing them in high-precision formats ensures accuracy but consumes significant memory bandwidth and storage space.
Quantization maps these higher-precision values to a lower-precision numeric format, such as 8-bit or 4-bit integers, using a scaling scheme that preserves as much of the original information as possible within the reduced precision. This mapping is not always a simple truncation; it often involves scaling factors to maintain the relative distribution of values.
Because each weight now takes up less memory, the overall model becomes smaller to store and load. This reduction in size means fewer bytes need to be moved between memory and the processor, which directly reduces latency. Additionally, computation using lower-precision numbers can also run faster on hardware that supports it, such as modern CPUs, GPUs, or specialized AI accelerators, because less data needs to be processed for each operation.
Post-training quantization vs quantization-aware training
The two most common approaches to applying model parameter compression differ in when the precision reduction occurs relative to the training process.
Post-training quantization applies the precision reduction to an already-trained model, without any further training, making it a fast and simple way to compress an existing model. You take a fully trained model, analyze the distribution of its weights, and convert them to lower precision. This is ideal for quick deployment when you have a pre-trained model and need to shrink it rapidly. However, because the model has not adapted to the lower precision, you may see a noticeable drop in accuracy, especially with very low bit-widths like 4-bit.
Quantization-aware training incorporates the effects of reduced precision during the training process itself, letting the model adapt to the lower precision as it learns, which typically preserves more accuracy at the same bit-width than post-training quantization, at the cost of requiring a training (or fine-tuning) run rather than a one-time conversion. During this phase, the training algorithm simulates the noise and rounding errors that quantization will introduce, allowing the model to adjust its weights to compensate. This usually results in a model that retains higher quality after conversion than post-training quantization achieves, though it requires more computational resources during the training phase.
Common formats and toolkits
To implement quantization effectively, developers rely on specific file formats and tools designed to handle lower-precision data efficiently.
GGUF/GGML is a file format and set of tools, used by the llama.cpp project, designed for running quantized large language models efficiently on consumer hardware such as laptops. GGUF (GGML Universal Format) is the current standard for storing quantized models, allowing for flexible loading and inference on various hardware architectures without needing complex dependencies.
GPTQ and AWQ (Activation-aware Weight Quantization) are quantization methods specifically designed to preserve model quality while reducing precision, each using a different approach to decide how to quantize weights with minimal accuracy loss. GPTQ uses second-order statistics to calibrate quantization parameters, while AWQ selectively preserves important weights based on their magnitude. These methods are often used for transformer-based models to maintain performance at low bit-widths.
Why it matters for small language models
Quantization is one of the key techniques, alongside knowledge distillation and pruning, used to make large models runnable on smaller hardware, including turning a model into a practical small language model (SLM) that can run on a phone, laptop, or other limited-hardware device. Without quantization, many modern models would simply not fit into the memory of consumer devices, making local deployment impossible.
A quantized version of a model can often run on hardware that would not have enough memory to hold the original, higher-precision version at all. This democratizes access to AI capabilities, allowing developers to build applications that run entirely on-device, improving privacy and reducing latency by eliminating the need for constant network calls to a cloud server. For inference on edge devices, this efficiency is critical.
Limitations
Reducing precision too aggressively can noticeably degrade a model’s accuracy or output quality, so there is a trade-off between how much a model is compressed and how much performance it retains. This is particularly true for complex reasoning tasks or languages with rich morphologies, where subtle distinctions in weight values matter.
Not all hardware supports every low-precision numeric format efficiently, so the speed benefits of quantization depend on whether the target hardware and software stack can actually take advantage of the reduced precision. For example, while 8-bit integers are widely supported, 4-bit or even lower precision formats may require specific instruction sets or custom kernels to realize their full speed potential. Additionally, quantization is just one part of the optimization puzzle; for an edge model to perform well, it often needs to be combined with other techniques like CPU optimization or kernel fusion.
FAQ
What is quantization in AI?
Quantization is a technique that reduces the numerical precision of a model’s weights and activations, such as converting 32-bit floats to 8-bit integers. This shrinks the model size and speeds up inference while typically causing a slight reduction in accuracy.
What is the difference between post-training quantization and quantization-aware training?
Post-training quantization converts a trained model to lower precision without further training, making it fast but potentially less accurate. Quantization-aware training simulates lower precision during the training process, allowing the model to adapt and usually preserving higher accuracy.
Does quantization work for all types of models?
Quantization is most commonly applied to transformer-based models like large language models, but it can be used for convolutional neural networks and other architectures. The effectiveness depends on the specific model architecture and the quantization method used.
Is quantization the same as pruning?
No. Quantization reduces the precision of the weights (e.g., from 32-bit to 8-bit), while pruning removes redundant or less important weights entirely. Both are model compression techniques, but they work differently and are often used together for maximum efficiency.