A user says “get ready for bed” to a wall panel. It replies “done.” The living-room light may still be on, the wrong bedroom may have been selected, or a one-time night-light setting may have become a standing rule. Adding a voice entry point has not improved control if the user can no longer tell what the system intended or what the devices actually did.
The first product decision is therefore not where to run a model. It is what the panel must let a person see and correct. Many existing MCU panels can remain useful if they can show proposed targets, parameters, device acknowledgments and failures while a controlled local service handles language interpretation. A new MCU board or a Linux panel becomes a candidate only when the actual display, audio, memory, interface or maintenance requirements exceed the current product.
This article uses a hypothetical home with lights in a living room and a bedroom. It is a design exercise, not a customer deployment or a benchmark. No target panel BOM, latency, cost, power or recognition measurements are available. Every hardware recommendation below needs to be checked against the product's own board, firmware and control interface.
Start with the action the person needs to understand
A conventional panel works well when the user knows which light to tap. “Get ready for bed” is different: does it turn off the living-room main light, dim the bedroom night light, or create a schedule for future nights? A chat page cannot resolve those ambiguities on its own. The panel's useful new job is to expose the choices that were previously hidden.
For the hypothetical request, a credible interaction has four visible moments:
- Preview the interpretation. Show “turn off the living-room main light and set the bedroom night light to 20%.” Make clear that this is a one-time action. If the room is uncertain, ask the user to choose it.
- Confirm in proportion to risk. A clear, low-risk lighting command may need little friction. A lock or heater cannot inherit the same generic “OK” behavior. Sensitive actions may require a separate trusted authorization path; a button drawn on a public panel is not an identity check.
- Report execution accurately. Show whether each device request was sent, acknowledged, timed out or remains unknown. A changed icon or a model's sentence is not a device acknowledgment.
- Handle partial failure. If the living-room light turns off but the bedroom device is offline, identify the failed item and offer a deliberate retry for that item. Do not replay the entire scene blindly.
These are interaction states, not a demand for four modal dialogs on every command. A deterministic touch shortcut should remain fast. Ambiguous, multi-device or persistent requests justify more preview and confirmation. Too much confirmation makes the panel worse than a switch; none at all turns a model guess into an unexamined device command. The useful dimensions are ambiguity, impact and reversibility—not simply whether an AI service was involved.
AI-generated illustration, not a target-product UI or field test. Icon color is not a device acknowledgment.
“Do it now” and “do it every night” are separate capabilities
“Set the night light to 20% tonight” changes the present state. “Set it to 20% whenever I get ready for bed” changes future behavior. Similar wording must not send both through the same commit path. A standing rule should expose its previous and proposed versions, affected devices, effective time and a way to restore the prior rule. A preview can show the intended future behavior, but saving a rule cannot undo a physical action that has already happened.
The screen is a good place to make that distinction visible. It is not the authority that decides who can change a rule. Shared rooms, guests and children make display-level confirmation inadequate as an authorization mechanism. The control service must validate write permissions and the allowed scope of sensitive actions independently of model wording and UI visibility.
Compare routes by performance, cost and failure mode
Before shopping for a new SoC, inventory the existing resolution and refresh target, touch and audio hardware, RAM and PSRAM, connectivity, control API, OTA path, device-state acknowledgments and offline requirements. Without that inventory, neither “an OTA update is enough” nor “we need a Linux panel” is an evidence-based conclusion.
The cost comparison must include more than processor price. Retaining a panel may avoid enclosure and certification changes but introduce hub discovery, two-sided updates and support complexity. Replacing an MCU may improve display or audio headroom but reopen power, wireless and firmware validation. A Linux panel may support more software yet add boot-time, patching and field-recovery obligations. Compare all three against the same user-facing response and failure targets, not against three unrelated demos.
| Route | When it is worth testing | Panel responsibility | Cost or failure mode to budget |
|---|---|---|---|
| Keep the MCU panel and connect a local AI hub | The current HMI can show a preview and state page; the protocol can evolve; audio hardware is sufficient or voice is not required | Touch, visible confirmation, device status and basic control; the model runs off-panel | Hub discovery, session recovery, offline behavior, API versioning and dual OTA |
| Replace the MCU panel or module, still using a hub | Display, audio or memory is insufficient, but an open-ended model does not need to run on the panel | Richer HMI, audio front end and local state cache | Board and enclosure work, power, thermal, certification and firmware migration |
| Use a Linux panel for some AI services | Complex media, multiple processes or on-panel inference is a real requirement, and the team can maintain an OS | HMI and selected inference; control authority remains separate | Startup, resource contention, security updates and fault isolation |
This is not a ranking from “basic” to “advanced.” If the goal is to show and correct a compound action, test the existing panel plus hub first. Offline open-ended conversation and a multimodal UI demand a different memory, audio and maintenance assessment. A panel without a microphone, playback reference or audio interface cannot gain a complete acoustic path through software alone. It may still deliver useful touch-based previews and configuration before voice is ready.
The ESP32-S3 RGB LCD documentation describes framebuffers in PSRAM, contention between display DMA and CPU access, and tearing or drift risks under particular configurations. That supports a test requirement, not a product performance claim: measure frame stability and memory on the target resolution under the actual animation and audio workload. A fluid development-board demo does not establish that an installed panel will remain fluid.
The ESP32-P4 getting-started documentation says the chip has no built-in Wi-Fi or Bluetooth and needs an external wireless SoC. If the existing product depends on a single-chip radio and an established certification path, selecting P4 changes the wireless design, board interfaces and maintenance chain. It is not merely swapping in a faster MCU.
ESP-Brookesia is an AIoT HMI framework and a useful candidate for evaluating interface components and agent integration. Its existence does not establish compatibility with a particular panel's drivers, audio path or OTA system. Pin the framework version and build a small prototype with a room view, compound-action preview and partial-failure feedback. Measure memory, response and recovery on the target hardware before treating it as a product route.
Keep a deterministic boundary between AI and control
A model may propose that “get ready for bed” means two lighting actions and explain the consequences. The control service should decide which devices are visible to this user, whether parameters are within bounds and whether execution is allowed in the current state. A universal tool accepting arbitrary device IDs and service names expands authority; a polished preview would only make that expansion visible after the fact.
Home Assistant's LLM script-tool guidance explains that exposed scripts become callable tools, and that descriptions and parameters affect tool use. Its Ollama integration documentation labels device control experimental and says sentence triggers are not integrated. Together these support testing a constrained set of actions in actual conversations. They do not prove that a ready-made configuration automatically routes fixed commands first and falls back to a model on failure. A product needing that behavior must design and test the routing separately.
For an existing panel, the minimum useful interface contract is more specific than “call the model API.” The panel sends a session ID, room context, selected target and cancellation request. The hub returns candidate actions, required confirmation level and an intelligible rejection reason. The control service reports each device request as accepted, confirmed, timed out or unknown. Cancellation does not roll back an action already executed. After reconnecting, the panel must query final state by request ID rather than infer completion from a local animation. If an API reports only that a command was sent, the UI may say “request sent,” not “light turned off.”
Proactive suggestions should come after reliable current actions. If the panel suggests an earlier lights-off rule, it should show the reason, the affected rule and a way to ignore or disable the suggestion. A few conversations must not silently broaden device-control scope. Avoiding one false success or one wrong-room action often has greater first-release value than generating ten new automations.
Use three validation rounds to decide where to invest
Round one: touch and state, without a model. Choose one compound scene. Test both devices succeeding, one offline, delayed state and user cancellation. If the present platform cannot display target identity and acknowledgments accurately, repair the control interface or HMI first. A new model cannot supply missing device facts.
Round two: interpretation without direct execution. Let the model propose targets and parameters. Allow the user to edit or cancel them; execution still uses a fixed action set. Test duplicate device names, wrong-room context, negation, and “this time” versus “from now on.” Keep the proposed action, final user confirmation, control-service decision and device acknowledgment together in the acceptance record. A chat screenshot alone is insufficient. If target selection remains unreliable, narrow the supported language or return to deterministic controls; confirmation dialogs are not an unlimited safety net.
Round three: persistent rules and proactive suggestions. Add rule versions, write authorization, audit and restoration. Inject AI-service failure, internet loss, hub loss and device loss. Separately verify that physical switches, basic touch functions and existing schedules still work under each failure. “The internet is down” does not imply that the local hub and model are healthy.
The route follows the evidence. If round one succeeds on the existing panel, defer board replacement; this does not mean the voice, memory and audio workload of round two has passed. If round two exposes a bottleneck, compare replacement MCU boards under the same workload. Put Linux in the formal design only if round three actually requires complex media or on-panel inference and the organization can maintain its OS. A product can stop after any round. The meaningful upgrade is one in which people can see what the system plans, what the devices did, and how to keep control when something fails.