Suppose a team wants local question answering on a Linux gateway, then similar capability on an RK3588 device and in a phone app. A tempting plan is to run four runtimes, rank their tokens/s, and standardize on the fastest. That comparison skips an earlier question: can the team's model become an artifact the target runtime can load? It also skips backend support and who will rebuild, package, and release the model when it changes.
Start with the delivery target. For a general Linux CPU or supported GPU and a compatible GGUF model, llama.cpp is a sensible first route to check. If a supported Rockchip NPU is a product requirement, verify the RKLLM conversion, Runtime, and driver combination. A team already maintaining PyTorch models and native mobile apps can evaluate ExecuTorch's export and backend path. A team prepared to compile and distribute target-specific model libraries can evaluate MLC LLM. These are not four interchangeable model players, and none has a universal speed rank independent of hardware, model, and version.
This guide uses public project documentation to compare delivery contracts, not benchmark results. We have not run a locked model, quantization, context, and prompt set on the target devices. It therefore makes no claim about decoding speed, time to first token, peak memory, power, or production reliability. Backend lists can change; confirm every route against the version and device you intend to ship.
Identify the artifact before comparing speed
The first difference is what leaves the build machine. llama.cpp commonly loads a compatible GGUF model and exposes CLI and server entry points. Its official README describes CPU, Metal, CUDA, Vulkan, and other backend paths. That breadth is useful for testing a local model on different general-purpose machines, but a backend name in a README does not prove support for a particular board or that every operation stays on its accelerator. Official llama.cpp README
RKLLM has a different chain: Rockchip's PC-side RKLLM-Toolkit converts and quantizes a model, while a board-side Runtime and RKNPU driver run the resulting artifact. Its official repository lists RK3588, RK3576, RK3562, and RV1126B families. The supported model and quantization options still need checking against the selected SDK and firmware. Having a GGUF file and a Linux-based Rockchip board does not mean that file will use the board's NPU through RKLLM. Rockchip RKLLM repository
ExecuTorch exports a PyTorch model into a .pte artifact for its edge Runtime. The LLM path also needs tokenizer and generation components, quantization decisions, and a suitable target backend. A .pte is not a universal file that will accelerate on every phone. After choosing XNNPACK, Vulkan, Core ML, or a specific NPU route, the team still has to confirm export, operator coverage, and behavior on actual devices. Its attraction to a PyTorch-and-app team is an explicit export and integration path, not a promised speedup. ExecuTorch LLM guide; backend overview
MLC LLM puts weight conversion, configuration, and compilation of a target-specific model library into the delivery chain. Its documentation describes explicit compilation and JIT compilation on some Python paths. An iOS or Android product must then decide whether weights ship with the app, download on first use, or update separately, and how those weights stay compatible with the library and app version. Renaming a GGUF or .pte does not create a usable MLC artifact. Compilation can be valuable, but maintaining it in CI is a real product cost. MLC compilation guide; library and weight packaging
| Delivery question | llama.cpp | RKLLM | ExecuTorch | MLC LLM |
|---|---|---|---|---|
| Typical artifact | Compatible GGUF | Converted RKLLM model | Target-exported .pte plus generation components |
Converted weights, config, and target model library |
| Device boundary to check | Actual CPU/GPU backend and model support | Rockchip chip plus Toolkit, Runtime, and driver versions | App platform, delegate, and operator coverage | Target compiler backend, runtime library, and packaging path |
| Often-missed maintenance | GGUF and quantization versions, service config | Model conversion and board software lockstep | Export regression, app API, backend compatibility | Per-target compilation, library and weight distribution |
Use the table to eliminate routes that cannot ship, not to score performance. A model-family name does not make four artifacts equivalent: quantization, tokenizer, context settings, and generation code can differ. If procurement asks for one unchanged model file across all devices, test whether that requirement itself is realistic. A product may need several validated artifacts from one model source, each recorded with its own target and build provenance.
Put hardware and update ownership on the same decision sheet
A Linux gateway prototype is not an NPU deployment
Consider a gateway that answers questions about equipment manuals on site. No NPU has been selected yet. The team knows only that the network may be unavailable, a local API is needed, and the model must work with the retrieval flow. Starting with a supported model and llama.cpp service can separate functional validation from later resource measurements. Once the hardware is fixed, measure CPU or a supported GPU backend on that target.
If the eventual product requires a Rockchip NPU, the prototype cannot simply be “moved to the NPU.” It needs an RKLLM conversion path, driver compatibility check, and answer-consistency tests. Whether that extra chain pays off is a question for the target board, not an inference from a desktop benchmark. The offline retrieval and evidence constraints remain separate; our equipment-manual RAG guide covers those application-level concerns.
A phone app owns the model release cycle
Now consider a team embedding offline summarization in Android and iOS apps. It already has PyTorch training and evaluation, and wants to release model and app versions together. An ExecuTorch minimum path is worth testing: export one supported model, assign ownership for tokenizer and generation code, and run it on target phones with CPU or a chosen delegate. If the team instead plans to build target GPU model libraries and can maintain the TVM/MLC compilation and packaging infrastructure, MLC LLM is another candidate. Both routes create app-size, first-download, offline-start, weight-update, and rollback questions. A support badge for “Android/iOS” does not settle those choices.
Even “the same model” can cease to be a comparable test object after different quantization or operator handling. Export can fail; a backend can fall back to CPU; or a model architecture may have to change. Record each candidate as a chain: model source/version → conversion/export version → quantization → target backend → Runtime/driver → tokenizer and generation config. A ticket that says only “use RKLLM” cannot identify which layer should be rolled back when an update fails.
For OTA products, attach that chain to the release record. Weights, Runtime, driver, and app interface may not be independently upgradeable. Replay a fixed question set on a small target-device cohort before widening rollout. Our edge AI OTA rollout guide addresses the broader release problem; here the point is to make artifact dependencies visible before choosing the runtime.
A tokens/s screenshot is not an architecture decision
Before a comparison, define “same conditions”: actual device, power and thermal conditions, model source and weights, acceptable quantization differences, tokenizer, prompt, context length, concurrency, and output limit. Then record cold start, time to first token, sustained generation, peak memory, temperature, power, and answer quality. If conversion changes operators or quantization, describe the difference instead of presenting the numbers as a pure runtime race. This is a proposed acceptance design, not a report of tests already run.
Three cheaper gates come first. Can conversion or export complete reliably and can the artifact load on the locked Runtime version? Is the intended backend really executing the work, without an unacceptable CPU fallback? Can the device return to a known-good model, Runtime, and configuration after failure? A fast demo does not make a production route sound if any of those three checks fails.
Include user experience in the measurement plan. A field manual assistant needs a timely answer with traceable sources. A mobile app needs a controlled first download and useful offline behavior. An industrial terminal may share memory with camera, UI, and networking tasks. Average decode speed cannot stand in for first response, memory peaks, or long-running thermal stability. Without a target device and locked versions, these are test requirements, not evidence that one route wins.
When to abandon the first candidate
If a GGUF runs on the Linux prototype but the product contract specifies a Rockchip NPU, do not promise that llama.cpp will use that NPU merely because it has several general backends. Verify the actual board's backend; if the official Rockchip chain is required, assess RKLLM separately. In the opposite situation—hardware still uncertain, models changing frequently, and no capacity to maintain conversion matrices—binding to RKLLM early can turn model exploration into a firmware compatibility project.
ExecuTorch is a concrete candidate for a mobile team with a PyTorch export and native-app workflow. If the target model cannot export reliably or important operators have no suitable delegate coverage, “mobile-friendly” is not a reason to keep investing. MLC LLM's multi-target compilation can be attractive, but without ownership of compiler, model-library, and weight-package versions it is not a low-maintenance shortcut. The team can validate the product task with a simpler route while hardware and update frequency are still fluid.
The output of this selection should be a short responsibility record: target device, shippable model artifact, intended backend, rollback version set, and owner of conversion and app integration. If a field remains unknown, run a minimal test on one target device and record the unsatisfied condition. The Ollama private-deployment guide is a starting point for a higher-level local-service question; this article is about the artifacts and device responsibilities beneath that service.
Sources and scope
- llama.cpp official README
- Rockchip RKLLM repository
- ExecuTorch LLM and backend documentation
- MLC LLM compilation and packaging
These sources support claims about documented interfaces and delivery paths. They do not establish target-device benchmark results. Before a production decision, recheck exact project versions, model licenses, target OS and backend constraints, and keep failed test cases and rollback outcomes with the release record.