Can an 8 GB Edge Device Run a 2 GB Model? Budget Memory at Peak Concurrency

“Eight gigabytes of RAM minus a 2 GB model file leaves six for conversations.” That subtraction cannot approve a device. The file is a storage artifact; inference also keeps weight pages resident, allocates KV cache and compute buffers, and competes with the operating system and any retrieval, audio, or display service. One successful short chat says little about two overlapping long ones.

You can obtain two useful numbers before touching a board: the exact bytes of a fixed model artifact and a theoretical KV calculation from its architecture. Runtime workspace, system use, allocation policy, and recovery after a peak remain measurements for the target device. The 2 GB file and 8 GB device here are a hypothetical purchasing question, not a tested board result.

Why “8 minus 2” mixes incompatible accounts

“This model needs 4 GB” could refer to storage, resident weights, a process peak, GPU memory, or the entire device. Ask which one. The useful question is who occupies physical memory under the same model, configuration, and workload, not whether two numbers on different datasheets can be subtracted.

The layers are conceptual, not proportional to measured capacity. The white area at the top is unallocated headroom, not another use of RAM. If the runtime allocates KV for a second session, that demand consumes some of the same headroom; the illustration establishes neither a fixed multiplier nor a device result.

These are diagnostic accounts, not necessarily additive measurements. A final capacity figure needs a consistent runtime configuration and observation window. If the only known value is the GGUF or other model-file size, you have screened storage, not signed off RAM.

A four-bit weight calculation is only a floor

Suppose a hypothetical model has three billion parameters and every parameter occupies exactly four bits. Its bare arithmetic weight floor is 3,000,000,000 × 4 ÷ 8 = 1,500,000,000 bytes, about 1.40 GiB. At eight bits the corresponding floor is about 2.79 GiB; at sixteen bits it is about 5.59 GiB. The calculation deliberately distinguishes decimal GB from binary GiB.

Real artifacts are not packed uniformly at that theoretical floor. Quantization blocks may carry scales or zero-points; some tensors may stay at higher precision; headers, alignment, embeddings, and output layers also consume space. Use the bytes of an artifact with a fixed hash, then inspect what the chosen runtime actually loads. “Q4” and “INT4” are not interchangeable byte-count promises across formats and backends.

More importantly, weight quantization does not shrink the operating system, the compute graph, or the KV cache by the same factor. The llama.cpp server reference exposes model-loading mode, context size, device offload, and K/V cache types as separate controls. If two measurements differ in several of those settings, the file format alone cannot explain the difference.

Memory mapping does not make RAM free

A runtime may map a model file rather than copying it into one obvious heap allocation. Pages accessed during inference can still become resident and may be shared between processes. A process's RSS can include shared pages, so RSS is not automatically the device's exclusive model cost. Linux documents RSS and proportional-set-size accounting in smaps_rollup. PSS helps describe shared mappings, but neither RSS nor PSS alone answers whether a separate GPU has enough memory or whether a device allocator can satisfy a large request.

For a target-device test, keep both process and system observations: available memory, any container limit, swap or OOM events, and device-memory figures where applicable. Record when each figure was captured. A healthy idle reading and an inference peak are different facts.

Where the 512 MiB KV estimate comes from

During autoregressive generation, the model reuses previously calculated keys and values instead of rebuilding them for every new token. The cache exists across attention layers and grows or is reserved according to the runtime's cache strategy. Hugging Face's cache explanation describes that per-layer K/V behavior. For ordinary attention with a uniform layout, an initial estimate is:

KV bytes ≈ 2 × attention layers
  × KV heads × dimensions per head
  × allocated tokens per session
  × bytes per K/V element
  × simultaneous independent sessions

The leading two represents keys and values. Use the number of key-value heads, not the number of query heads printed elsewhere in a configuration. Grouped-query and multi-query attention may use fewer KV heads than query heads. If keys and values use different data types, calculate their terms separately and add them. A sliding-window layer, a hybrid architecture, or shared state across layers needs a model-specific calculation instead of this uniform expression.

A calculation, not a board measurement

Assume 32 standard attention layers, eight KV heads, 128 dimensions per head, and two bytes per K or V element. If a runtime allocates 4,096 tokens for one session, the uniform estimate is 2 × 32 × 8 × 128 × 4,096 × 2 = 536,870,912 bytes, or 512 MiB. Two independently allocated sessions at that same limit would have a 1,024 MiB theoretical KV baseline. Doubling the per-session allocation to 8,192 tokens doubles that term if the other assumptions remain fixed.

The 512 MiB figure is not a measurement of a commercial model or board. “4,096 tokens” here is an assumed allocation, not a number of visible characters. System instructions, retrieved passages, user input, and reserved output all count toward a request's token budget. Some runtimes reserve the entire configured context; others grow or reuse cache space, allocate blocks, or carry batching overhead. A 100-token chat cannot establish the peak for a 4,096-token configuration, let alone two overlapping long chats.

Two sessions do not always mean “multiply by two.” Under the same hypothetical architecture, one session allocated 4,096 tokens and another allocated 2,048 tokens would have a 512 MiB plus 256 MiB, or 768 MiB, theoretical KV baseline if their caches are independently allocated at those lengths. A runtime reserving two full 4,096-token slots would not obey that 768 MiB figure. Record each slot's effective allocation and the runtime's reported allocation; “two users online” is not a memory measurement. Sharing the model's weights does not imply that two different conversation histories share their KV.

Cache quantization and offloading move the trade-off

Lower-precision K/V may reduce the nominal cache-data term, but scales, residual storage, conversion buffers, and latency can change the real saving. Offloading cache from a GPU to the CPU changes where memory is consumed and adds transfer work; it does not create free device capacity. Hugging Face's cache strategies describe both quantization and offloading as memory/performance choices. The llama.cpp server options likewise let an operator choose K and V cache types separately.

When comparing two runs, record the weight artifact, K and V types, context allocation, session count, offload setting, and runtime version together. A result labeled only “4-bit” is ambiguous: it might describe weights while the cache stays at sixteen bits.

Turn 4,096 tokens into an actual request

Allocated tokens are not visible characters. Suppose the longest business input takes 3,000 tokens, the answer allowance is 800, and instructions and templates consume another 200. That is already 4,000. A 4,096-token context leaves little room for a longer retrieved passage or tokenizer variation; an extra 150 tokens can cause truncation before an out-of-memory failure. Raising the context to 8,192 may avoid truncation but can make some runtimes reserve more KV memory.

That is why a 100-token demo cannot approve an 8 GB product. Separate a short request, the longest allowed single request, and requests that genuinely overlap. If the product searches local manuals or drives a display, leave those services enabled during the concurrent run. Multiplying a single-request peak by a thread count is not a concurrency test: weights may be shared, KV allocation may be reserved or reused, and a runtime may silently queue work.

Likewise, “two users online” does not mean two full contexts are active. Define a concurrency workload that another engineer can reproduce—for example, two longest inputs overlapping during prompt processing. If a product intentionally serializes long requests to save memory, document the wait time and timeout behavior as part of acceptance. A memory reduction that breaks response-time requirements is not a free win.

Keep the unmeasured cells blank

Fix the product workload before treating any estimate as a requirement: longest actual input, output-token allowance, concurrent users, companion retrieval or audio services, and acceptable failures and latency. Then fix the model artifact hash, tokenizer, architecture configuration, runtime and backend versions, context limit, and K/V type. Without those inputs, a range of candidates is reasonable; “8 GB is enough” is not.

A working budget should show four stages side by side: static estimate, loaded idle device, longest single request, and maximum intended concurrency. The static stage can contain measured artifact bytes and a calculated KV baseline. Mark compute workspace and system use “to measure” rather than inventing a safety percentage. Observe the loaded stage with all normal services running.

Copy this blank worksheet for each candidate configuration. Enter values, units, and capture times, and attach the model-file hash, runtime version, and raw logs. Leave cells blank until measured; a static calculation must not masquerade as a device observation. Record both the device peak and MemAvailable in the final row so process RSS is not mistaken for the whole-system answer.

The process-total RSS/PSS can already contain mappings, KV, and buffers attributed to that process. The detailed rows explain a peak; they are not separate charges to add together. File bytes and RSS/PSS use different accounting boundaries too. Keep effective input/output tokens, allocated context, truly overlapping sessions, container limits, swap/OOM, and failed requests on the same record. Capture a timestamp for each number so MemAvailable and the reported peak refer to the same moment.

The measurement sequence is short enough to repeat. Boot the target image with its normal services and record idle MemAvailable. Load the artifact identified by a hash and preserve the runtime's effective weight, context, KV, and workspace settings. Run the longest single request, then genuinely overlapping requests, then observe recovery over several rounds. Useful Linux fields include /proc/meminfo MemAvailable and the inference process's smaps_rollup RSS/PSS; record container current usage and limits if containerized, and device-memory counters when the runtime provides them. If the runtime omits a field, write “not reported,” not a number borrowed from the model file.

Suppose all you have today is an 8 GB device specification, a 2 GB model artifact, and the hypothetical 512 MiB KV calculation above. The static column can hold artifact bytes and a formula. The next three columns must remain blank. That record says storage has a candidate and cache has a theoretical starting point. It does not say roughly 5.5 GB remains: resident pages, buffers, services, container limits, and actual allocation policy have not been measured on one accounting basis.

Once the first target-device log arrives, an unexplained difference between theoretical KV, runtime-reported allocation, and the system peak is worth investigating. It may reflect a longer reserved context, batching buffers, another process, or a different memory device. Do not “fix” the worksheet by tuning a multiplier until numbers match. A missing or surprising field tells you which log to collect next.

Two comparable rows fix the same device, operating-system image, runtime version, and request set while changing one candidate configuration. Comparing a 3B and a 7B model requires reading each model's own layer and KV-head counts; the first model's cache-per-token constant does not transfer. Comparing two weight formats requires checking that the tokenizer, context, and K/V precision did not change in the background. There are no such paired device runs behind this article, so it supplies a test design, not a ranking of models for 8 GB hardware.

Locate the failure before buying more memory

“The model won't run” can mean that input was rejected before inference, that loading or generation ran out of memory, or that only two overlapping requests fail. The branches below begin with an observable failure stage rather than assuming every failure calls for more RAM.

Preserve the last successful action before a failure. If loading completed but the first request failed, inspect input processing and its allocation. If short requests work but a long one is truncated, inspect the tokenizer and output allowance; more RAM may not address a context-policy error. If only concurrency causes OOM, inspect overlapping time ranges, per-session KV, and container or device limits. If the first round succeeds and later rounds deteriorate, inspect cache release and queued work. Change one variable and rerun the original load; otherwise a new success does not show what fixed the original problem.

Interpret counters against the memory topology. On unified-memory hardware, process RSS, a GPU API counter, and system availability may describe some of the same physical pages, so adding them can double-count. With discrete VRAM, system RAM can remain available after device memory is exhausted. A container can also hit its own limit while the host still reports headroom. One OOM line without the request shape and effective runtime settings does not prove that the product needs “at least 16 GB.”

Even if quantizing weights or shortening context makes a configuration fit, either change can damage answers, time to first token, or the manual passage the product must retain. After capacity screening, use a task-relevant answer set and the separate TTFT, TPOT, memory, and power validation guide to judge the result.

When can you call 8 GB enough?

Only for a fixed artifact, effective runtime configuration, and target workload: the longest request and intended concurrency must complete, and the device peak, failures, swap/OOM record, and recovery must meet the product's requirements. File bytes and theoretical KV can screen candidates; they cannot approve capacity. If the product must search long manuals while serving a display and multiple users, and reducing context or concurrency would break that requirement, compare a larger-memory device or another deployment approach using the edge-LLM hardware selection guide.

There are no target-board runs, paired model scales, or paired quantization formats behind this article. The three-billion-parameter, 512 MiB KV, and 2 GB/8 GB figures are worked examples, not performance results for an RK3588, GPU, or AIHub device. A useful purchasing conclusion today may be: “The theoretical account is complete; the whole-device concurrency peak still needs measurement.” That is more informative than an unqualified “it runs.”

Sources

• Hugging Face Transformers: how caching works
• Hugging Face Transformers: cache strategies
• llama.cpp server options
• Linux kernel documentation: /proc and smaps_rollup