Taalai's AI Hardware Co.
Reference

AI Hardware Lab

The mechanics behind every number on this site — what determines whether a piece of hardware can run a given model, and why. Diagrams, not just paragraphs.

GPU memory Parameters Quantization Watts and power Rack servers Inference vs. training Why some models need multiple GPUs

GPU memory

A GPU has its own dedicated memory, separate from your computer's system RAM. Before a GPU can run any part of a model, that model's weights have to be loaded into this memory. If the model doesn't fit, the GPU cannot run it — there is no fallback to system RAM for the parts that don't fit; the process fails or has to be restructured (e.g. split across more GPUs).

Live — this fills up as you change the numbers below
GPU memory capacity Model weights +20% working room Fits → runs
⚙ Try it yourself — the diagram above updates live
PrecisioniHow many bits are used to store each parameter. 16-bit is the common default and the safest estimate; 8-bit and 4-bit are "quantized" versions that use fewer bits per parameter, which shrinks memory needs roughly in proportion (8-bit ≈ half the memory, 4-bit ≈ a quarter) but can reduce output quality somewhat. See the Quantization section below for the full explanation.

GPU memory comes in two families used across this catalog: GDDR, used on desktop and workstation cards (cheaper per gigabyte, still fast, less total bandwidth), and HBM ("High Bandwidth Memory"), stacked directly on the chip in datacenter GPUs like the H100 and H200 (higher bandwidth, higher cost).

In simple terms A GPU is like a chef, and its memory is like the chef's kitchen counter. Before the chef can start cooking, every ingredient — the whole AI model — has to already be laid out on the counter. If the counter's too small, some ingredients just won't fit, and the chef can't cook that recipe at all, no matter how skilled they are.

Parameters

A model's "parameters" are the individual numeric values it learned during training — roughly, its knowledge encoded as numbers. A 70-billion-parameter model has 70 billion of these values. Each one has to be stored somewhere, and at full (unquantized) precision each parameter takes about 2 bytes, which is why parameter count translates almost directly into a memory requirement: roughly 1 GB of memory per billion parameters at the common 16-bit precision most open-weight models ship in.

Some models — DeepSeek-V3.2 and Kimi K2.6 in our model advisor, for example — use a "mixture of experts" design, where only a fraction of the total parameters activate for any given token. That reduces compute per token, but all the experts still have to be resident in memory in case they're needed, so memory requirements are still based on the total parameter count, not the active count.

In simple terms A model's parameters are like the number of tiny switches inside its "brain" — the more switches it has, the more it can learn, but each switch also needs its own little bit of storage space. A 70-billion-parameter model just means there are 70 billion of these switches that all need somewhere to live.

Quantization

Quantization reduces the precision used to store each parameter — for example, from 16-bit down to 8-bit or 4-bit — which shrinks the memory footprint roughly in proportion (8-bit is roughly half the memory of 16-bit; 4-bit roughly a quarter). This is a real, widely used technique, and it's why you'll see the same model described with very different memory requirements in different places.

We calculate memory math on this site at 16-bit precision (1 GB per billion parameters) because it's the safest, most conservative baseline and the format most open-weight releases ship in by default. If you plan to run a quantized version of a model, its real memory requirement will be lower than what we show — often significantly. Quantization can also reduce output quality to some degree; how much depends on the model and quantization method.

In simple terms Quantization is like writing a phone number with fewer digits by rounding it — you save space, but lose a little precision. Doing this to a model shrinks how much memory it needs, the same way a rounded number takes less room to write down than the exact one.

Watts and power

Watts measure the rate of electricity draw, not compute capability. A higher-watt GPU doesn't hold more memory or automatically run models better — it typically means more raw compute (faster), a larger electricity bill, and more heat that has to be removed from the room. Power matters for two practical reasons: your electricity cost, and whether your facility's power and cooling infrastructure can actually support the hardware.

A single RTX 5090 draws about as much power as a couple of hair dryers running continuously. An 8-GPU server draws roughly as much as several houses. A full GB300 NVL72 rack draws roughly 100x an average home's continuous load — which is why full racks are only deployed in facilities built for that load, not offices or homes.

In simple terms Watts tell you how much electricity something drinks, kind of like how many glasses of juice someone drinks. A phone charger takes a small sip. A giant AI computer rack gulps down as much juice as a hundred houses combined — that's why these machines need special rooms with extra power and extra cooling, the same way a huge animal needs a bigger cage and a lot more food than a hamster does.
standard 120V circuit (1920 W safe continuous) Desktop card RTX 5090: 575 W — about 0.5x an average home's continuous draw 8-GPU server 8x H100 SXM server: 5600 W — about 4.7x an average home's continuous draw Full rack GB300 NVL72 rack: 135,000 W — about 112x an average home's continuous draw

Bar height is not to true linear scale — a real rack draws roughly 200× a desktop card, compressed here so all three stay visible. Hover, tap, or tab to a bar for the exact real wattage.

Hover or tap a bar above to see its real power draw.

Rack servers

Datacenter GPU servers are built to mount in a standard 19-inch rack, measured in "rack units" (U) — 1U is about 1.75 inches of vertical space. An 8-GPU HGX-based server is typically 4U–8U depending on the integrator and cooling design. A full rack is usually 42U–48U total, so a single 8U server takes up roughly a sixth of a rack; a fully populated rack like the GB300 NVL72 is designed and sold as one 48U unit, not assembled piecemeal from individual servers.

Rack-mounted systems assume rack-scale infrastructure already exists: power distribution units rated for the load, hot/cold aisle or liquid cooling, and network cabling — not just an electrical outlet.

In simple terms Think of a datacenter rack like a chest of drawers — each "U" is one drawer-sized slot. A big server might take up several drawers at once, and a full rack is the whole chest, built to hold as many computer "drawers" as will neatly fit, stacked one on top of another.
⚙ How many servers fit in a rack?

Inference vs. training

Inference is running a model to produce output — the model's weights are fixed, and the GPU just computes a forward pass. This is comparatively cheap: it needs enough memory to hold the model plus a working buffer (what we call the 20% working room), and a single GPU or a small cluster is often enough.

Training (including fine-tuning) updates the model's weights, which means the GPU also has to hold gradients and optimizer state alongside the weights themselves — commonly 3-4x the memory of the weights alone, sometimes more. This is why training hardware recommendations lean toward multi-GPU servers with NVLink rather than single cards, even for models that would fit in a single card's memory for inference alone.

In simple terms Inference is like reading a book that's already written — you just need room to hold it open. Training is like WRITING the book yourself — you need a much bigger desk for all your notes, drafts, and crossed-out pages, not just space for the final, finished book.

Why some models need multiple GPUs

When a model's memory requirement exceeds any single GPU's capacity, it has to be split across multiple GPUs — a technique generally called model parallelism. For that to work efficiently, the GPUs need a fast, direct connection to share data with minimal delay. NVIDIA's datacenter GPUs use NVLink for this within a server; connecting multiple whole servers into a larger cluster additionally requires InfiniBand networking between them.

In simple terms NVLink is like kids standing right next to each other, so they can pass notes instantly. InfiniBand is like kids in different rooms passing notes through a mail slot — it still works, but it's much slower than just handing a note to the person standing beside you.
Server A Server B NVLink mesh inside Server A: every GPU talks to every other GPU directly, at very high bandwidth and low latency. NVLink mesh inside Server B: every GPU talks to every other GPU directly, at very high bandwidth and low latency. InfiniBand: the networking layer connecting separate servers into one cluster. Much lower bandwidth than NVLink inside a server — notice only one packet at a time crosses here, versus several moving at once inside each server's mesh — which is why NVLink-connected GPUs act like one giant GPU but InfiniBand-connected servers don't. InfiniBand NVLink mesh (dense, low-latency)

Hover, tap, or tab to a server's mesh or the InfiniBand link to see what it means. The moving dots show relative traffic density, not literal packets.

This is also why we only build "clusters" out of hardware designed for it (see the clusterable flag on each product) — plugging several desktop cards into one PC over ordinary PCIe technically works, but is far slower for a single large model and isn't how real multi-GPU deployments are built.

Try the model advisor

Get matched to a build