Batch Size Memory Calculator
Estimate the activation memory a training batch consumes from batch size, sequence length, hidden size and layer count — the term that scales with batch and blows up GPU memory.
How it works
Activation memory ≈ batch × sequence length × hidden size × layers × bytes × k (k ≈ 10 for stored transformer activations)
During training the forward pass stores activations for the backward pass, and that memory scales linearly with batch size — which is why doubling the batch is what usually triggers an out-of-memory error, not the weights. This estimate uses a rough multiplier for the several activation tensors kept per transformer layer; your exact figure depends on architecture, attention implementation and whether activation checkpointing is on. Treat it as an order-of-magnitude planning number: if it dwarfs your GPU memory, lower the batch, shorten sequences, or enable gradient checkpointing to trade compute for memory.
Worked example
Batch 16, sequence 2048, hidden 4096, 32 layers in bf16 (2 bytes): per sample ≈ 2048 × 4096 × 32 × 2 × 10 ≈ 5.37 GB, so a batch of 16 needs about 85.9 GB of activations — before the weights and optimizer state, which is why it must be split across GPUs or checkpointed.
Frequently asked questions
Why does batch size, not model size, cause out-of-memory errors?
Weights and optimizer state are fixed regardless of batch, but activations scale linearly with batch size and sequence length. So the batch is the knob that most often tips you over the GPU’s memory limit — halving it roughly halves activation memory.
How accurate is this estimate?
It is a rough order-of-magnitude figure using a fixed activation multiplier. Real usage varies with the attention kernel (FlashAttention stores far less), activation checkpointing, and framework overhead. Profile your actual run for exact numbers.
What is activation checkpointing?
A technique that discards most activations during the forward pass and recomputes them during the backward pass. It can cut activation memory dramatically (often to the square root of the layer count) at the cost of extra compute — the main lever when you are memory-bound.
Related calculators
Educational estimate — real costs and memory depend on your provider, framework, model architecture and pricing, which change often. Verify against your own bills and profiling before committing budget.