Ever since LLMs became widely available, I have been curious about running them locally. But once you start reading about it, you quickly end up in a rabbit hole. There are many different models out there, with names like "Qwen3-30B", terms like "full precision", and plenty of other jargon.

This post is a simple guide to some of the most common terms. We will start with the model name itself, then move into what parameters and tokens are, and finally look at what precision and quantization mean in practice.

But what does it all actually mean? What is a token? What is precision? And what does 30B mean? And so on...

Parameters

Let's start with the easiest part.

The B in a model name stands for billions of parameters. For example:

  • Qwen3-8B: 8 billion parameters
  • Qwen3-30B: 30 billion parameters
  • Qwen3-80B: 80 billion parameters

Ok, great. But what are parameters?

Parameters are the internal values a model uses to represent what it has learned during training. You can think of them as the model's countless tiny settings or weights. During training, these values are adjusted over and over again until the model becomes good at predicting the next token in a sequence.

A parameter is not a fact stored in plain text, like "Paris is the capital of France". Instead, parameters are more like compressed patterns. They capture relationships in language, code, facts, style, reasoning patterns, and much more.

So when people say that a model has 8B or 30B parameters, they mean that it has 8 or 30 billion learned values shaping how it responds. In general, more parameters means the model can represent more complex patterns, but it also requires more memory and more computation to run.

That is why parameter count matters so much when people compare local models. It tells you something about both the model's potential capability and the hardware it may require.

  • 1-4B: Small models for simple chat and lightweight tasks.
  • 7-14B: Good all-round models.
  • 30-40B: Very capable for many use cases.
  • 70-80B: Close to GPT-4-class for some tasks.
  • 200B+: Flagship-size models.

Tokens

Since models are trained to predict the next token, it also helps to understand what a token is.

A token is a small piece of text that the model processes at a time. It is not always a full word. Sometimes a token is a whole word, sometimes part of a word, and sometimes punctuation.

For example, the sentence:

I like local LLMs.

might be split into tokens similar to:

"I" " like" " local" " LLM" "s" "."

The exact split depends on the model, but the important idea is that models read and generate tokens, not words.

This matters for two reasons:

  • Speed is often measured in tokens per second.
  • Context size is measured in tokens, for example 8K, 32K, or 128K tokens.

So when a model "writes", it is really predicting one token at a time, again and again, based on the tokens that came before it.

Now that we know what parameters are and what tokens are, we can move on to the next big question: how these parameters are stored in memory.

Precision

Precision describes how the model's parameters are stored in memory.

Original models are typically trained or stored in formats such as:

  • FP32 (32-bit)
  • BF16/FP16 (16-bit)

When we run them locally, we often compress them using quantization. For example:

  • FP16: about 100% of the original memory footprint
  • 8-bit: about 50%
  • 4-bit: about 25%

So for a 32B model, the rough memory usage might look like this:

  • FP16: ~ 64 GB
  • 8-bit: ~ 32 GB
  • 4-bit: ~ 16 GB

This is the reason quantization matters so much for local AI. It can make the difference between a model fitting on your machine or not fitting at all.

What this means in practice

So what does this mean when we put it all together?

In practice, it mostly comes down to how large a model you can fit into your available RAM or VRAM.

Let's say your GPU has about 30 GB of available VRAM. Then, roughly speaking:

  • An 11B model in FP16 would fit.
  • A 27B model in 8-bit would fit.
  • A 58B model in 4-bit would fit.

In other words, you could say: "I can run a 58B model on my 30 GB GPU if I quantize it to 4-bit."

That does not mean the model becomes useless. It means the model is compressed, usually with some quality loss, but often not enough to outweigh the benefit of using a larger model.

So what's the best option here then?

Interestingly, a larger model with 4-bit quantization is often better than a much smaller model in FP16.

So in many cases, a 32B Q4 model can perform better than a 14B FP16 model.

That is why local LLM discussions often focus so much on the trade-off between parameter count and quantization level. People are really asking: "What is the best model I can fit into my available RAM or VRAM?"

Quantization names

With that said, you may also see names like Q4_K_M, Q4_K_L, Q5_K_M, or Q6_K.

Err, what does that mean then?

When you load a GGUF model in Ollama, you may see filenames like:

Qwen3-30B-Q4_K_M.gguf
Qwen3-30B-Q5_K_M.gguf
Qwen3-30B-Q8_0.gguf

These names indicate different quantization levels:

  • Q2: low quality, very small.
  • Q3: acceptable quality, still very compressed.
  • Q4: often the best balance between quality and memory use.
  • Q5: very good quality, but larger.
  • Q6: close to the original model.
  • Q8: very close to the original, but still quantized.

Q4 is often the most popular choice because it is a good balance. If you have memory to spare, you might prefer Q5 or Q6. Q8 is still compressed, but much less so than Q4.

The extra parts, such as _K_M, are more specific variants of the quantization method. For most beginners, the important thing is the first part: Q4, Q5, Q6, and so on. That gives you the main idea of how compressed the model is.

MoE models

Also, one last confusing term: MoE. Qwen3, for instance, has models like Qwen3-32B and Qwen3-30B-A3B.

The latter, Qwen3-30B-A3B, is an MoE model, short for "Mixture of Experts".

That means the model has 30 billion total parameters, but only about 3 billion are active for any given token.

In other words, it can behave more like a smaller model in terms of active compute, while still benefiting from a much larger total parameter count.

Summary

So if we zoom out, the big picture is this:

  • The model name tells you roughly how large the model is.
  • Parameters tell you how much the model has learned.
  • Tokens are the pieces of text the model reads and writes.
  • Precision and quantization tell you how much memory the model needs.
  • MoE models are a special case where the total size and active size are not the same thing.

Once you understand those pieces, names like Qwen3-30B-Q4_K_M become much less mysterious.