A supplier's “8 GB RAM, 6 TOPS NPU” line does not establish that a field device can answer questions about a long service manual offline. The memory figure says nothing about the OS, application, and model sharing that capacity. The accelerator figure says nothing about whether the target model runs on that NPU's software stack. Specify the model, quantization, context, and concurrent sessions first; budget memory and verify the backend next; compare acceleration and whole-device power only after those gates pass. Otherwise, a short demo can work while the deployed workload crashes or slows under longer inputs and sustained heat.
This is a method for eliminating unsuitable candidates, not a benchmark ranking of RK3588, GPU boards, and CPU-only devices. Without the same model, inputs, software versions, and enclosure conditions, a claim that one board is faster, cheaper, or more power-efficient would be an untested claim.
Define the job before choosing a processor
“Run a 7B model” is still too vague for procurement. An engineering lead needs the expected input length, output length, concurrent requests, any retrieval or speech service, and what must still work without network access. Two devices may both load the weight file, yet face very different memory demand when one must retain a long conversation or serve multiple users. If those conditions are decided after purchase, testing may only reveal that the chosen hardware cannot meet the product requirement.
Consider a control panel that accepts brief commands and a service assistant that reads several pages of a repair manual. Both can be called local-LLM products. The second also needs room for retrieved passages, conversation history, indexing, and application services. If the device displays video or processes audio at the same time, model startup alone says even less about the complete product.
If the job is manual-based question answering, include retrieval and document updates on the workload card. Offline RAG for equipment manuals addresses those system conditions; a model-only memory budget does not replace them.
Make a workload card that another engineer can rerun
Record the exact model and quantized artifact, runtime version, maximum input and output tokens, concurrency limit, background services, required offline actions, acceptable waiting time, and enclosure constraints. Avoid filling this card with guessed tokens-per-second numbers. A change to the model, quantization, context, or number of parallel sessions invalidates a direct comparison with an earlier run.
The card should also name the failure path. Will the UI ask for a shorter document, queue the request, switch to a smaller model, or fall back to a network service where permitted? Hardware that “just fits” without a defined fallback is fragile when manuals grow or more users arrive.
Budget runtime memory, not only model storage
A model fitting on eMMC proves storage capacity, not inference capacity. During inference the runtime maps or loads weights in a supported form; attention uses a KV cache whose demand depends on the request and configuration. Computation buffers, input processing, the OS, and the application also consume memory. A practical first-pass budget has five parts: resident weights, KV cache, runtime workspace, OS and application reserve, and incremental memory for concurrency or other device tasks. If a proposal reports only the model-file size, the RAM decision is not ready.
The official Hugging Face KV-cache documentation notes that the cache can consume substantial memory for long contexts. Cache offloading and quantization trade memory against latency; they are not free capacity. The llama.cpp server reference exposes separate settings for context, parallel slots, cache types, cache offloading, and model-layer placement. Changing those settings changes where memory is used. The same GGUF file therefore does not imply the same runtime peak on every configuration.
Quantization solves one part of the budget
Reducing weight precision usually lowers weight storage. It does not automatically shrink the KV cache, runtime buffers, and OS footprint by the same proportion. A particular quantization format must also be supported by the chosen runtime and accelerator. A 4-bit artifact that loads but fails the product's answer-quality checks, or moves a bottleneck into CPU-memory transfers, is not a successful hardware decision.
Do not treat “8 GB” as a universal safe threshold. Boards with that nominal capacity may reserve different amounts for the OS, display stack, GPU, NPU, and always-on services. Start the target image with all resident services, record available memory, and then run cold starts, the longest input, repeated requests, and maximum planned concurrency. Measure both process and system peaks. Leave room for logging, updates, recovery, and later model versions; determine that margin from the actual device and risk tolerance rather than copying another board's number.
Verify the backend before discussing TOPS
TOPS is a capability figure for a category of operations, not end-to-end token latency for a named model. Operators, quantization, driver, kernel, and runtime must form an executable combination. Rockchip's RKLLM project, for example, publishes supported platforms and model families. It also describes a PC-side Toolkit conversion step followed by a board-side Runtime, C API, and NPU driver. “RK3588 supported” is enough to put a board on the evaluation list; it does not mean arbitrary weights can be copied onto it and served unchanged.
A general-purpose runtime and a vendor NPU path solve different maintenance problems. One may make a common model format deployable across more devices. The other may require converted artifacts and a frozen toolkit, driver, and system-image combination. If a team cannot maintain that conversion and image lifecycle, a large advertised accelerator figure may turn every model upgrade into a compatibility project. When the product genuinely requires a particular NPU, take the actual target model through conversion, board startup, and critical operator checks before committing to volume purchase.
PyTorch's ExecuTorch LLM guidance likewise treats export, deployment, and target backend as concrete steps. Framework support for on-device LLMs is not a blanket performance guarantee across SoCs. Ask for the precise model, runtime, driver, and OS-image versions that form the proposed deployment. Ask who will revalidate that set after an update. Those answers matter more to delivery than a bare TOPS comparison.
If the software stack is still undecided, narrow it using the edge LLM runtime and model-artifact selection guide. This article answers the next question: whether the chosen hardware can carry that route.
Sustained operation is a different test from model startup
Even after the model fits and the backend starts, whole-device behavior remains unproven. Generation repeatedly accesses weights and cache. Larger models, longer contexts, and more sessions can increase pressure on memory channels and data movement. The CPU, GPU, or NPU may also share resources with display, camera, and network work. A single idle-board demo cannot establish response time under the actual business load. Published specifications cannot be rearranged into a trustworthy user-experience number here; the target system must be measured.
Test short commands separately from long-manual questions, and single sessions separately from the planned concurrency ceiling. Capture time to first token, intervals between later tokens, process and system memory peaks, clock frequency, temperature, and whole-device power. Continue until thermal behavior has stabilized, then examine throttling and latency drift. For a continuously operating device, a one-minute cold-start demonstration validates a functional path, not sustained service.
Advertised memory bandwidth can eliminate an obviously unsuitable candidate, but it cannot be converted directly into tokens per second. If the target board exposes memory-controller counters, keep the model, context, and concurrency fixed and record memory traffic both alone and alongside display, camera, or other product tasks; compare those readings with time to first token and later token intervals. Without usable counters, repeat a fixed workload before and after adding those concurrent tasks and compare latency and clocks, but do not present the difference as a precise bandwidth-utilization figure. If capacity and backend compatibility pass yet latency drifts substantially under concurrent work, investigate memory-path and scheduling contention before blaming NPU compute capacity.
Enclosure and power design can reverse an early impression. An open development board on a desk may cool adequately while the same electronics in a fanless cabinet, near a wall, or next to another heat source may not retain the same clocks. If the final product also renders a display, captures speech, or operates as a gateway, keep those tasks active during testing. Only then do the measured memory, bandwidth, and thermal demands resemble the shipped device.
The sequence below is a set of validation gates. If capacity fails, debating accelerator TOPS is premature. If the target model does not work on the backend, advertised compute cannot rescue that incompatibility.
Eliminate deployment routes in a useful order
| Route | First gate | Cost that remains after it passes |
|---|---|---|
| CPU-focused Linux board | Does the quantized artifact run with reserve at the target context? | A smaller model, shorter context, or longer wait may be necessary; measure it. |
| Edge board with GPU | Can weights, cache, and other applications coexist in its real memory layout? | Power, cooling, driver, and GPU allocation require whole-device validation. |
| Vendor NPU board | Can the target model be converted and run with matching operators, quantization, driver, and image versions? | Conversion and version maintenance remain part of the product. |
| Cloud or hybrid route | Do offline, privacy, cost, and latency requirements permit a network path? | Network dependence, data governance, and continuing service cost must be owned. |
The table is not a speed ranking. If the weight and runtime peak do not fit, shrink the model or change hardware first. If the dedicated backend does not support the model, more TOPS do not fix the mismatch. If sustained response is unacceptable, compare cooling, quantization, accelerator backend, and hybrid deployment under the same test conditions. Change one important variable at a time so an apparent improvement has an identifiable cause.
When a local LLM is the wrong requirement
A device that only maps fixed commands or retrieves a few structured states may be simpler to maintain with rules, a smaller classifier, or a permitted cloud service. Frequent model upgrades are also a poor fit for a board with little memory and driver headroom. If connectivity is acceptable, the data can be handled within applicable requirements, and local sustained performance misses the target, include hybrid and cloud architectures in the same review rather than excluding them with a slogan.
When offline operation is essential, not every task must use one large model. Limit the device-side guarantee to the intents, retrieval, or short answers that must work without a network. Provide a clear failure message or a permitted cloud fallback for harder requests. This reduces the workload the product promises to sustain; it does not disguise a weak hardware estimate.
Keep evidence that survives the purchasing meeting
A credible selection record does not require a glossy TOPS chart. It must let another engineer rerun the decision: model and quantized artifact versions, runtime and driver versions, system image, context and concurrency settings, resident services, power and enclosure conditions, test inputs, peak memory, and sustained thermal-response records. A missing measurement should say “not tested,” not be filled with a vendor support matrix.
The public documentation supports a screening order: place the actual workload first, then verify memory, backend, and sustained whole-device constraints. A final purchase decision still needs the same workload on the target board in the target enclosure. Until then, retaining two testable candidates is safer than declaring that a particular NPU or RAM figure is “enough.”