Deploying RKLLM on RK3588: Pin Versions, Convert the Model, and Validate On-Device Inference

Copying Hugging Face weights onto an RK3588 board does not make the model run on its NPU. Rockchip's RKLLM path has two distinct sides: an x86-64 development machine loads, quantizes, and exports a .rkllm artifact with RKLLM-Toolkit; the board loads that artifact through the RKLLM Runtime C/C++ API, which in turn depends on the RKNPU driver. Pin the model, Toolkit, Runtime, driver, and OS image as one versioned set before debating launch commands or speed. Without that record, a successful export and a failed board initialization are difficult to diagnose as parts of the same deployment. Rockchip repository

This guide follows Rockchip's release-v1.3.1 and its published examples. It is an execution and acceptance plan, not a benchmark of a production device. Commands require your actual model path, cross-compiler, and target-board image. We do not have conversion logs, inference logs, peak-memory readings, or long-run data from a specified RK3588 board; therefore, this article does not claim tokens per second, time to first token, power draw, or compatibility with a vendor's production image.

The cover and inline hardware photographs are generated scene illustrations, not photos of the target board or evidence that it has run this model.

Record the version set before conversion

A common time sink is to download a .rkllm file from one machine and try it on a board running somebody else's old image. The filename may contain RK3588, but it does not identify the upstream model revision, quantization dataset, context setting, Toolkit release, board-side librkllmrt.so, or kernel driver. When initialization fails, “the model won't open” is too vague: the artifact could be incomplete, the user-space Runtime could be incompatible, or the board could be short of memory.

Start with a deployment record containing the model repository and immutable revision; tokenizer and chat template; quantization dataset and version; Toolkit wheel; export-script commit; target platform; SHA-256 of the .rkllm file; Runtime library; RKNPU driver; OS image; and board revision. If the application adds its own prompt template and sampling settings, pin those too. Treat this as one testable deployment unit, not a shopping list. Changing the model without changing the Runtime, or flashing an image that replaces a driver, creates a new set that needs validation.

The official release contains Toolkit wheels, Linux and Android Runtime packages, C API headers, examples, driver archives, and doc/Rockchip_RKLLM_SDK_CN_1.3.1.pdf. Their presence shows what Rockchip shipped in v1.3.1; it does not show what a board vendor integrated into its image. Ask the image maintainer for the kernel build, NPU driver identity, user-space library source, and upgrade method. If those cannot be obtained, pause the idea of “just replacing the model” and obtain a reproducible image or controlled flashing package first. Official release, repository tree

There is a subtle version trap here. The rkllm_api_demo README specifies rkllm-toolkit>=1.3.0 and rkllm-runtime>=1.3.0 as minimums for that example. That is not a promise that every new or old exported model will be interchangeable with every Runtime. For a reproducible starting point, take the Toolkit, Runtime, headers, and example from the same release, then regression-test upgrades as a set. Do not read a minimum version as permission to upgrade only the Toolkit indefinitely. API demo README

This image illustrates the handoff discipline; it is not evidence of a tested board model or inference result.

Convert on the development machine in diagnosable stages

Check that the input model fits the locked toolchain

Rockchip's example uses DeepSeek-R1-Distill-Qwen-1.5B to illustrate the flow. It is an example, not a recommendation for every product or a guarantee of acceptable response time on every RK3588 memory configuration. If you substitute another model, check whether the locked release supports its architecture, whether weights and tokenizer share a revision, whether the license permits your use, and whether the chat template or custom configuration needs attention. Otherwise, a conversion exception may reflect an input/script mismatch rather than a general limit of RK3588.

Build quantization data for the actual task

The first official step is not to run export_rkllm.py immediately. It is to produce data_quant.json with the example data-generation script. That script loads a model and creates input/target pairs for calibration; its own comments say the data should reflect the model and use case. The bundled questions mix mathematics, code, and everyday dialogue. They are not a ready-made quality baseline for industrial-device maintenance, Chinese customer support, or controlled equipment commands.

For a maintenance assistant, include the vocabulary and input lengths that real users will bring, plus refusal and error-boundary cases. Version the dataset separately from the model artifact so a changed dataset triggers a new test. The sample uses trust_remote_code=True: review the selected model repository's code and license, and run unfamiliar model code in an isolated development environment rather than on a production build host. Quantization data script

Inspect the contents of data_quant.json after generation; the existence of a file is not a valid calibration check. The official script catches some generation errors and writes an empty target in those cases. Count total entries, empty targets, and invalid records; then check whether the actual content covers the intended tasks before calling build. Otherwise, a bad calibration file may masquerade as a model or NPU problem.

Do not use the identical questions for both calibration and final acceptance. Calibration should represent the product's common input distribution; a held-out acceptance set should ask whether the exported model still recognizes domain terms, follows output constraints, and avoids invented answers when evidence is missing. Reusing one set can make a quantization change look reassuring in the lab and fail on a customer's wording. There is no universal accuracy percentage to quote here: accuracy depends on the model, data, and task, and needs a task-specific test on the intended device.

Export an artifact that can be traced back to its inputs

The v1.3.1 example calls RKLLM().load_huggingface(...), then build(do_quantization=True, quantized_dtype="W8A8", target_platform="RK3588", num_npu_core=3, max_context=4096, dataset="./data_quant.json", ...), and finally export_rkllm(...) to write a .rkllm file. These are example values, not a claim that W8A8, three NPU cores, or a 4096-token context is optimal for your model and board. Official export script

A larger max_context changes memory demands. Quantization affects accuracy, memory, and the resulting artifact; setting three cores does not replace a board resource check. First reproduce one supported path, then change one variable at a time and tie each result to the exported file's hash.

The command order below assumes a development machine with a working CUDA environment, as the official script's main path specifies load_huggingface(..., device='cuda') and sets CUDA_VISIBLE_DEVICES='0'. The latter selects a GPU; it does not add CUDA capability to a CPU-only machine. Before running, change modelpath in export_rkllm.py to your local model directory and check the quantization file, Python environment, and available memory. For a CPU-only host, first confirm that your model and Toolkit support that route, change the script's device to cpu, and reassess conversion time and host RAM. Do not run the CUDA sample unchanged. We have not run these commands on a target device. Official export script

cd examples/rkllm_api_demo/export
python generate_data_quant.py -m /path/to/your/model
python export_rkllm.py

When export fails, investigate the stage rather than immediately lowering quantization precision. At load time, inspect the model directory, weights format, tokenizer, and dependencies. At build time, inspect architecture support, calibration data, operators, and target-platform parameters. At export time, retain the full log, free disk space, and output path. Every failed run should record the Toolkit wheel, script commit, input revision, parameters, and first meaningful error. Without those, a retry may silently test a different version set.

On the RK3588 board, prove loading before judging answers

Compile for the target ABI and load the right libraries

After conversion, the board demo must be built for the target architecture. The Linux build-linux.sh contains an AArch64 cross-compiler path from the example environment; it will not automatically exist on your development machine. Point GCC_COMPILER_PATH to the actual toolchain, verify that its ABI fits the board image, and ensure the program loads librkllmrt.so from the selected release. Copying only the executable and omitting the Runtime or libomp.so can produce a dynamic-library error that says nothing about NPU model compatibility. Rockchip also mentions missing libomp.so on some platforms. Linux build script, official repository

Once compiled, inspect the executable and lib directory under install/demo_Linux_aarch64. Do not copy the example adb push commands onto a product that does not use ADB; the transfer method should follow that device's flashing and access policy.

Initialization is a start, not product acceptance

The board example sets LD_LIBRARY_PATH=./lib and runs ./llm_demo <model-path> 2048 4096. Its C++ code creates default parameters with rkllm_createDefaultParam(), supplies model_path, max_new_tokens, and max_context_len, calls rkllm_init(), and receives output through a callback. The values 2048 and 4096 are demo parameters, not default generation limits for every application. C API demo source

For a first acceptance run, check that initialization succeeds and one fixed short prompt completes; then test a longer context. The official single-process demo cannot establish application concurrency. After a service layer is integrated, design a separate load test covering concurrent requests and resource contention. If even the short prompt cannot initialize, inspect the .rkllm hash, Runtime path, and driver before rewriting business prompts. Demo run instructions

One sentence of generated text is still not a delivery verdict. Test the same .rkllm after a cold boot, after hours of operation, and while the board's camera or other business services are active. That exposes resource contention, long-context memory pressure, and recovery behavior. Save model hash, image, library, and driver identifiers for each run; otherwise, a changed answer could be caused by input, temperature, sampling randomness, or version drift. If the final enclosure and power arrangement do not exist yet, label the result as development-board evidence, not production acceptance.

The example performance flow includes fix_freq_rk3588.sh. Fixed clocks can help compare configurations, but they are not the production default or proof that a sealed device can sustain that frequency. Keep at least two test groups: default system policy, which reflects deployment constraints, and controlled performance testing, which helps isolate model/configuration differences. Report any measured result together with board, cooling, power supply, clock policy, image, input length, and generation limit, or another team cannot reproduce it. Rockchip run instructions

Locate the failed stage before changing parameters

There are three useful boundaries: did the development machine produce the intended .rkllm; can the board process load its libraries and initialize that model; and does inference meet the product's task and resource limits under target load? Calling every failure “model incompatibility” encourages repeated, expensive conversion while ignoring a driver or memory problem.

The important junction is where the .rkllm artifact meets the board image and driver at Runtime load. If loading fails, compare versions and logs on both sides before deciding to reconvert the model.

The table is an investigation order, not a one-to-one diagnosis. Initialization can fail for version, file, or resource reasons; board logs and the pinned record narrow it down. If the export script has been changed, reproduce from the smallest same-release baseline: use the original sample on a CUDA host, or retain only the necessary device='cpu' adaptation on a CPU-only host, then restore other changes one at a time. If the board image changed, verify the older artifact on a recoverable test device before expanding an upgrade. Do not overwrite Runtime libraries across the fleet and discover that older models can no longer roll back.

Four checks between a demo answer and a deliverable application

Artifact traceability. Put the model source and license, input revision, calibration dataset, export script, Toolkit, .rkllm SHA-256, Runtime, driver, and image into the release record. Only then can an incident owner decide whether to roll back the model, library, or entire image. Avoid logging complete customer prompts or sensitive data; use a redacted, reproducible test set. The official repository supplies process entry points, not your version ledger.

Functional correctness. Use fixed, reproducible prompts to test short answers, long context, domain terms, stop conditions, and malformed input. If a product can operate equipment, never execute a model answer directly as a control command. Identity, authorization, parameter bounds, confirmation, and audit belong in the application layer. A successful answer to an official demo puzzle does not prove suitability for an industrial task. Compare before and after quantization using the same held-out test set, not one favorable sample.

Resources and stability. On the intended board and enclosure, measure cold start, model load, time to first token, sustained generation rate, peak process memory, NPU/CPU load, temperature, throttling, and behavior alongside other services. The repository provides RKLLM_LOG_LEVEL=1, CPU/NPU observation scripts, and clock-control tools as starting points, but it provides neither this product's duty cycle nor its pass thresholds. Define thresholds from acceptable waiting time, concurrency, thermal conditions, and offline requirements before testing. For a comparable TTFT, TPOT, memory, and power measurement protocol, see our edge LLM benchmark guide. Official performance-method entry points

Upgrade and rollback. Give the model, Runtime, driver, image, and application prompt template explicit versions and coupling rules. Keep at least one prior combination that can be restored on a test device. Acceptance must cover not only a new version starting, but also a rollback, re-entry into service after failure, and avoidance of corrupted artifacts after interruption. An image update that replaces the kernel driver is not “just shipping a new model file.” These are proposed acceptance checks, not a claim that we have completed OTA or endurance tests.

When RKLLM should not be the default path

If hardware is not fixed, the model changes every week, or nobody owns the board image and driver, binding the project to RKLLM immediately can turn model exploration into a version-compatibility project. First test whether a model solves the business task on a general-purpose Linux device; then decide whether NPU conversion is worth productizing. Our edge LLM runtime selection guide compares delivery artifacts across runtimes. This article deliberately starts after RK3588 and RKLLM have been selected.

Conversely, RKLLM has a clear pilot role if the product is already committed to RK3588, needs offline inference, and the team controls coordinated releases of the image, Runtime, and model. The next step is not a larger performance claim; it is a small supported model on one specified board, with a recorded loop of conversion, initialization, fixed prompts, resource measurements, and rollback. Until that record exists, call the capability a deployment plan, not a validated device feature.

Sources and evidence limits

• Rockchip RKLLM release-v1.3.1: release identity, packages, and changes.
• Rockchip official repository: Toolkit, board Runtime, drivers, and observation tools.
• Rockchip API demo: sample model, conversion, build, and run sequence.
• Rockchip export script and board-side C++ example: parameter and API locations.

We verified public release materials and example code, not conversion, flashing, inference, or performance on a target board. Instructions to “record,” “check,” and “accept” describe a proposed sequence, not a completed test. SDK support, model compatibility, and vendor images can change. Recheck the exact version set and run target-device validation before procurement or rollout.