Fitting AI in your pocket with quantization

Every time I tried to run a decent open-weights model on my own laptop, I hit the same wall. The model was "only" 7 or 13 billion parameters, and my GPU still choked on it. Either it wouldn't load, or it loaded and left me no memory for anything else.

The fix that actually works isn't a bigger GPU. It's quantization.

If you've heard the word thrown around — "we quantized it to 4-bit," "int8 inference," "GGUF" — but never actually done it yourself, this post is for you.

In this tutorial, we're going to:

• Break down what quantization actually does to a model's weights, in plain terms

• Walk through the difference between downcasting, linear quantization, and blockwise quantization

• Quantize a real Hugging Face model in a few lines of code using Hugging Face's Quanto library

• Check exactly how much size and speed we get back, and what we give up for it

By the end, you'll have a working quantized model on disk, and a mental model good enough to make sensible quantization decisions on your own projects.

---

What Problem Is Quantization Actually Solving?

Let's be specific about what "the model won't fit" looks like in practice.

A model's weights are just numbers — millions or billions of them, one per learned connection in the network. By default, most models store those numbers as float32: 32 bits, or 4 bytes, per parameter. Do the math on a 7B-parameter model and you're looking at roughly 28 GB just to hold the weights, before you've generated a single token.

That memory requirement is the actual bottleneck. It's not compute in the sense of "not enough FLOPS" — modern consumer GPUs have plenty of raw throughput for small-to-mid models. It's that the weights don't fit in VRAM in the first place.

Quantization attacks this directly: it stores each weight using fewer bits. Drop from float32 to int8, and that same 7B model shrinks to around 7 GB. Suddenly it fits on a card that couldn't hold it before, loads faster, and often runs faster too, since moving less data around means less time waiting on memory bandwidth.

The catch — and it's an important one — is that fewer bits means less precision per number. You're deliberately throwing away information. The entire discipline of quantization is about doing that throwing-away in a way that the model barely notices.

---

The Core Idea: Fewer Bits, Same Structure

Picture a single weight matrix in a model — just a grid of decimal numbers, learned during training. Quantization takes that matrix and maps it onto a smaller set of possible values, usually integers, then remembers how to convert back.

Think of it like resizing a large photo down to a thumbnail. You lose some fine detail, the file gets dramatically smaller, and for most purposes the thumbnail is still perfectly recognizable. Push the compression too far, though, and the picture starts to look wrong. Quantization has the same trade-off curve — there's a point where it's essentially free, and a point past that where the model's outputs visibly degrade.

The gap between a weight's original value and its quantized-then-restored value is called quantization error. Almost everything in this field is really about minimizing that error for a given bit budget.

There are a few distinct ways to actually do the bit-reduction, and they're not interchangeable.

Downcasting

The simplest approach: just change the storage type directly, without any clever remapping. Take a float32 tensor and cast it straight to a lower-precision float type like bfloat16 (Google's "Brain Float," which trades precision for keeping float32's wide dynamic range).

This works fine down to bfloat16 — models tolerate it well because the format was specifically designed to preserve range. But push downcasting further, to something like int8, and it falls apart. Integer types don't have the same handling of the number's range and distribution that floats do, so naive downcasting to int8 introduces enough error that model quality visibly drops. In practice, downcasting is a good speed trick for the float16/bfloat16 tier, and a poor strategy for anything more aggressive.

Linear Quantization

This is the workhorse technique behind most int8 and int4 quantization you'll see in the wild. Instead of a direct cast, it computes a mapping:

1. Find the min and max values in the tensor

2. Compute a scale factor that stretches that float range to fit the target integer range (0–255 for uint8, for example)

3. Compute a zero-point, so that 0.0 in float-land lands on a sensible integer

4. Quantize every value using that scale and zero-point

5. At inference time, dequantize back to a higher-precision value on the fly, run the actual computation, then discard the higher-precision copy

The key trick is that you only ever store the compact integer weights plus two small numbers (scale and zero-point) per tensor. The expensive floats never sit in memory — they're reconstructed just long enough to compute with, then thrown away again. That's where the memory savings actually come from.

Blockwise Quantization

Linear quantization applies one scale and zero-point to an entire tensor. That's fine when the weights are fairly evenly distributed, but real weight matrices often aren't — you'll get a few outlier values stretching the range, forcing everything else into a narrower effective precision band.

Blockwise quantization fixes this by chopping each tensor into smaller blocks and computing a separate scale/zero-point per block. It costs a bit more metadata, but it captures local variation in the weight distribution far more accurately, which matters a lot for tensors with uneven, "spiky" value distributions — which describes most trained neural network weights.

One More Distinction Worth Knowing: PTQ vs. QAT

You can apply quantization at two different points in a model's life:

• Post-Training Quantization (PTQ) — take an already-trained model and quantize it afterward, with no retraining involved. This is what we're doing in this tutorial. It's fast and requires no training infrastructure.

• Quantization-Aware Training (QAT) — simulate quantization noise during training, so the model actually learns weights that are robust to it. This produces better accuracy at very low bit-widths, but it's far more expensive, since you need the full training pipeline.

For most people quantizing an existing open-weights model, PTQ is the right tool. QAT is really only worth it if you're training your own model from scratch, or fine-tuning specifically for a quantized deployment target.

---

Let's Quantize a Real Model

Theory's useful, but the fastest way to build real intuition here is to just do it. We'll use Quanto, Hugging Face's quantization library, because it wraps all the scale/zero-point/calibration bookkeeping into two function calls.

Step 1 — Install What We Need

pip install transformers accelerate quanto torch

Step 2 — Load a Model and Check Its Baseline Size

We'll use a small model here — Pythia 410M — so this runs comfortably on a laptop CPU. The same code scales to bigger models; you'll just want a GPU for those.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "EleutherAI/pythia-410m"
model = AutoModelForCausalLM.from_pretrained(model_name, low_cpu_mem_usage=True)
tokenizer = AutoTokenizer.from_pretrained(model_name)

Before touching quantization, it's worth confirming the model actually works, and logging its starting size so we have something to compare against later:

def get_model_size_gb(model):
    total_bytes = sum(p.numel() * p.element_size() for p in model.parameters())
    return total_bytes / 1e9

print(f"Original model size: {get_model_size_gb(model):.3f} GB")

text = "Once upon a time, there was a"
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=10)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

In my experience, always run this sanity check before quantizing anything. If the base model is already misbehaving, quantization isn't the thing to debug first.

Step 3 — Quantize

Quanto's API splits quantization into two explicit steps, and understanding why matters. quantize() swaps the model's linear layers for quantization-aware versions, but doesn't touch the actual weight values yet:

from optimum.quanto import quantize, freeze
import torch

quantize(model, weights=torch.int8, activations=None)

At this point, if you print a weight tensor, you'll notice it still looks like the original float values — nothing's actually been converted. That's intentional; it gives you a chance to inspect the model structure before committing to the conversion.

Step 4 — Freeze

freeze() is the step that actually applies the quantization — it computes the scale and zero-point values and converts the stored weights to the target integer type:

freeze(model)

print(f"Quantized model size: {get_model_size_gb(model):.3f} GB")

On Pythia 410M, going from float32 to int8 weights typically gets you down to roughly a third of the original size — the linear layers, which dominate parameter count in most transformer models, are exactly what got converted.

Step 5 — Confirm It Still Works

A smaller file that produces garbage isn't a win. Run the same generation call again:

outputs = model.generate(**inputs, max_new_tokens=10)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

One inference isn't a rigorous eval — don't take a single generated sentence as proof of preserved quality. If you're shipping a quantized model to production, run it against a held-out eval set and compare perplexity or task accuracy before and after. For a quick gut-check during experimentation, though, seeing coherent output here is a reasonable first signal.

---

What You Actually Give Up

It's worth being honest about the trade-off curve instead of pretending quantization is free:

• int8 weight-only quantization is close to lossless for most models. This is the safe default if you just need the memory savings.

• int4 and below start showing measurable quality degradation, especially on tasks requiring precise reasoning or exact factual recall. Techniques like blockwise quantization and calibration help a lot here, but they don't eliminate the trade-off.

• Activation quantization (as opposed to weight-only) is harder to get right, because activations are dynamic — their range depends on the actual input, not just the trained weights — so it needs proper calibration against representative data rather than a one-time computation.

If you're deploying to production, the right process is: quantize, then evaluate on your actual task, then decide if the size/speed win is worth whatever quality delta you measured. Skipping the evaluation step is the most common mistake I see.

---

Where to Take This Next

A few natural extensions once you've got this working:

1. Try 4-bit quantization. Libraries like bitsandbytes or GGUF-based tooling (llama.cpp) push further than int8, which is where you'll really feel the laptop-friendly size difference on larger models.

2. Quantize a model you actually want to deploy. Swap Pythia 410M for a 7B or 13B instruction-tuned model, and re-run the size and quality checks. This is where quantization stops being an exercise and starts being a real infrastructure decision.

3. Build a proper eval harness. Instead of eyeballing one generated sentence, run the quantized and unquantized models against a small benchmark set and compare scores side by side. That's the only way to know if you've actually preserved performance or just gotten lucky on one prompt.

---

Conclusion

Quantization solves a very specific, very real problem: models are too large to fit in the memory you have, and full precision is mostly overkill for inference anyway. By converting weights to lower-precision integer types — carefully, using scale and zero-point mapping rather than a naive cast — you can shrink a model to a fraction of its original size while keeping performance close to the original.

We walked through the three main flavors — downcasting, linear quantization, and blockwise quantization — and then quantized a real Hugging Face model end-to-end using Quanto, going from a full float32 model down to int8 weights in about ten lines of code.

The bigger takeaway: quantization isn't a niche research technique anymore. It's a standard step in the deployment pipeline for anyone running LLMs outside a data center, and it's genuinely this accessible to add to your own workflow.

#llm #quantization #machinelearning #huggingface #python #ai