“Turn off the main light” is transcribed perfectly, but the bedroom light goes out instead of the living-room light. “Get ready for bed” works with one phrasing and fails with another. A television says something that sounds like a command, and now the household wonders whether a device might act on it. Those are not three versions of the same speech-recognition problem. A larger local model cannot solve them as one problem.
Separate audio capture, transcription, room and device matching, and permission to act. Predictable controls can keep a deterministic path; an LLM may interpret flexible phrasing, but an execution layer outside the model must constrain targets, parameter ranges, and changes to persistent rules. Local inference gives you more control over part of the data path. It does not automatically improve recognition or prevent an unauthorized action.
We will use an illustrative household, not a customer deployment: a main light in each of two rooms, an ESP32 voice satellite in the living room, Home Assistant for devices and scripts, a local Chinese ASR service, and an optional local LLM. The configuration below is a virtual lab. No household accuracy or latency result has been measured.
The words are right. Why did the wrong light turn off?
Start with the same request through two entrances: type “turn off the living-room main light” into Assist, then speak it through the microphone. If the typed request also fails, inspect entity names, exposed entities, intent matching, and the execution path before tuning microphone gain. If typing succeeds and speaking fails, investigate wake detection, captured audio, and ASR. A successful text test does not prove that the microphone works; it simply removes several downstream guesses from the immediate search.
For a failed request, retain the transcript, selected target, and action result together. A screenshot of “I didn't understand” does not show whether the assistant heard the wrong words, picked the wrong entity, or issued a correct action to an unavailable device. Home Assistant's voice troubleshooting guide and automation traces are useful at different points; correlate them to the same request.
| Symptom | Check first | Likely place to change |
|---|---|---|
| Listening never starts, or the TV repeatedly wakes the satellite | Wake events, active listening periods, similar-sounding TV audio | Wake model, threshold, placement, or push-to-talk entry |
| The transcript drops “don't” or changes a device name | Captured audio, clipped beginnings/endings, language setting | Capture, VAD, ASR model and task configuration |
| The transcript is right but a different room is selected | Satellite area, entity names, aliases, exposure list | Areas, entity registry and intent matching |
| The target is right but nothing changes | Action trace, device connection, state freshness | Integration and device execution chain |
| The device changes but the assistant says it failed | Tool return, state readback, TTS timeout | Result protocol and feedback path |
Word error rate alone is a poor measure of this example. If “main light” is transcribed correctly and room selection is wrong, a better ASR model will not fix the target. Conversely, if ASR drops “don't” from “don't turn off the main light,” a perfectly functioning executor can still perform the opposite action. Mark negation, room names, and action verbs separately in the test set so that many easy utterances do not conceal the consequential mistakes.
Improve the local Chinese voice path in the right order
Preserve the speaker's voice before tuning suppression
In a living room, the first problem may be the mixture of the speaker, television, fan, and the satellite's own loudspeaker—not the size of the recognition model. Fix the satellite's position, speaking distance, and orientation. Listen to separate recordings in a quiet room, with the TV playing, and while the satellite speaks. Check for clipping, missing first words, and truncated negation. Change one setting at a time; changing model, gain, and placement together makes the result impossible to attribute.

Illustrative scene only: television audio and placement can affect what the microphone receives. This is not an acoustic test result.
When the satellite hears its own response, distinguish acoustic echo cancellation (AEC) from general noise suppression. AEC needs a reference to the device's playback signal. Espressif's ESP-SR audio front-end documentation distinguishes microphone channels from the playback reference. Enabling an AEC flag without the correct reference, channel mapping, and timing does not prove that full-duplex listening works. An independent television is not the satellite's known playback reference either.
ESPHome's voice assistant parameters include noise suppression, automatic gain, and volume controls. They are not evidence that a particular firmware has connected an ESP-SR playback-reference pipeline. Verify where processing occurs in the deployed firmware. Raising gain also raises background noise; aggressive suppression can erase a quiet speaker. There is no universal best combination for every microphone and room.
Voice activity detection (VAD) determines the boundaries of an utterance. An early end can cut off a slow speaker; a late one adds waiting and may capture TV speech. Keep separate samples for pauses, children, and short commands. If reliable listening during playback is not yet available, pause reception while the device speaks and resume afterward. That loses natural interruption but is more controlled than recognizing the device's own answer as another command. For buffer and transport issues, see the ESPHome voice-satellite troubleshooting guide.
“Local speech recognition” is not one interchangeable capability
Home Assistant distinguishes Speech-to-Phrase, built around a known command set, from Whisper, which accepts more open expression. Check the target language before copying a fast English-language demonstration into a Chinese household. Standard Mandarin, regional accents, dialects, children's voices, and code-switching are different test conditions. Support for one does not establish support for the others. See Home Assistant's local voice setup guidance.
In the Speech-to-Phrase repository and migration notes reviewed on 4 November 2026, the original repository is retained through the Kaldi-based v1.4.3 release, while newer development has moved to OHF-Voice/apps. Its listed served languages do not include Chinese. Do not treat Speech-to-Phrase as an out-of-the-box default for Chinese control. Recheck the actual release and phrase coverage if that support changes.
If you evaluate Whisper for Chinese, choose a multilingual model rather than an English-only .en model. Compare explicit Chinese selection with automatic detection on your fixed-Chinese task set. Test mixed Chinese and English separately; locking Chinese is not automatically better for those utterances. The task should preserve the original-language transcript, not translate a device command into English and hand it to a control layer with only Chinese entity names. See the Whisper model documentation.
Model size also changes the latency budget. A quick isolated transcription does not establish acceptable response times when ASR, TTS, and an LLM share one host or several rooms speak at once. Long waits may encourage repeat commands. For frequent short requests such as turning off a light, deterministic intent matching after transcription is a useful comparison path; not every utterance needs an extra LLM round trip.
Only configure device-name hints, hotwords, or vocabularies when the actual ASR backend supports them. Whisper, Whisper server wrappers, and other Chinese ASR engines do not expose one universal hint interface. A hint cannot serve as reliable protection against a lost negation. If a crucial action word is absent or interpretations conflict, ask for clarification rather than having an LLM guess the opposite intent from household habits.
Two rooms share the name “main light”
In the illustrative household, use light.living_main and light.bedroom_main with readable names such as “living-room main light” and “bedroom main light.” Both may have the short alias “main light.” Assign the living-room satellite to the living-room area, then test a fully qualified command before an unqualified one. Home Assistant's alias documentation covers entity, area, and floor aliases and how a satellite's area can help matching.
An alias addresses name-to-target matching; it does not train ASR. If audio becomes an incorrect transcript, adding an alias is not proof that recognition will improve. If the transcript is correct but the bedroom target is chosen, investigate area association, duplicate names, and the actual selected entity instead of continuing to tune audio.
Room context can also fail: the satellite may move without its registered area changing; a phone request may lack the same room source; or two devices in one area may have the same short name. Test the complete room name first. If the product needs to ask when several candidates match, verify that refusal-and-clarification behavior through the actual entry point or implement candidate checking. Do not claim every Assist route provides it by default.
Expose a small set of low-risk entities first. Making every device and script available just to improve “turn off all lights” matching enlarges the control surface. Entity exposure lets an object enter Assist's control path; it is not a complete identity and authorization system for guests, children, and owners. See Assist device exposure.
“Get ready for bed”: make parameters flexible, not execution unlimited
A bedtime routine might turn off the living-room main light and dim a bedroom night light. The useful configurable parts are the targets, allowed brightness, enabling conditions, and feedback. They do not require the LLM to choose a fresh device list every time. In particular, avoid a generic tool that accepts arbitrary entity_id and service names from a model.
These are alternative entrances, not a built-in automatic fixed-command-first router. The script's checks, not the model's answer, determine whether an action can proceed.
There are two different entrances. A fixed utterance can trigger a Home Assistant custom-sentence automation; more open phrasing can lead an LLM to select an audited script tool. They may call the same script, but writing a custom sentence does not automatically make it work inside an Ollama conversation. The current Ollama integration says it does not directly integrate sentence triggers and treats device control as experimental. A fixed-command-first router with an LLM fallback needs a verified routing path or additional implementation, not just an arrow on a diagram. See also custom sentences.
A virtual-state configuration, not a real lighting installation
The accompanying local_voice_policy_lab.yaml is a complete teaching package. Two helpers stand in for living-room on/off state and bedroom brightness. Separate policy helpers hold an allow switch, a guest restriction switch, and a brightness target from 5 to 35. This is an illustrative range, not a lighting standard or a measured model capability.
The package was checked for YAML parsing and static constraints against Home Assistant Core 2026.10.4 interfaces. It has not been loaded into Home Assistant or exercised with speech. The satellite firmware, microphone, ASR/TTS weights, and model-service image are not a tested deployment combination. First back up an isolated test instance, place the package under its packages directory, and merge the following line into the existing homeassistant: section—do not create a second top-level section or paste the entire package into the automation editor. See Home Assistant packages.
homeassistant:
packages: !include_dir_named packages
The script's essential boundary is shown below, preserving the Chinese labels and responses of the accompanying package rather than presenting a different configuration as the same lab. It accepts no device or action name; the two virtual targets are fixed, and brightness is read from a separate helper. The full attachment also defines the helpers and matches the Chinese lab phrases 准备睡觉演练 and 执行睡前灯光演练; the English “get ready for bed” wording here is not itself a configured trigger.
script:
local_voice_bedtime_lab:
alias: 睡前灯光演练
description: 只更新两项虚拟灯光状态,不操作真实设备,不保存或修改策略
mode: single
sequence:
- variables:
level: "{{ states('input_number.local_voice_night_level') }}"
- choose:
- conditions:
- condition: template
value_template: >-
{{ is_state('input_boolean.local_voice_allow_bedtime', 'on')
and is_state('input_boolean.local_voice_guest_mode', 'off')
and is_number(level)
and 5 <= (level | float(-1)) <= 35
and states('input_boolean.lab_living_main') in ['on', 'off']
and is_number(states('input_number.lab_bedroom_level')) }}
sequence:
- action: input_boolean.turn_off
target:
entity_id: input_boolean.lab_living_main
- action: input_number.set_value
target:
entity_id: input_number.lab_bedroom_level
data:
value: "{{ level | float }}"
- variables:
result:
status: simulated
message: 已更新虚拟灯光状态,未操作真实设备
- stop: 虚拟演练步骤结束
response_variable: result
default:
- variables:
result:
status: rejected
message: 未执行演练,请检查允许开关、访客模式与实体参数状态
- stop: 策略或状态检查未通过
response_variable: result
The initial configuration has the allow switch off and the guest restriction on, so the first call should be rejected. A tester must explicitly enable the routine, disable the guest restriction, and provide valid helper states before the action branch can run. The condition checks off explicitly; an unknown or unavailable state does not silently become permission. Brightness must be numeric and within range. The reject branch makes no action call.
Here “guest mode” is only a global restriction switch. It has not identified who is speaking. Do not expose the policy helpers as voice-editable entities, or the assistant could disable the restriction and then call the script. Hiding the controls in a UI is not an API authorization boundary either. If policies differ by user, establish a real authenticated subject and verify the deployment's permission model, potentially through a separate service that enforces authorization independently.
The complete lab calls script.local_voice_bedtime_lab directly to receive its response_variable, then sets the conversational response. Replacing that call with a fire-and-forget variant does not preserve the same result. On the model route, inspect the actual tool call and return; the assistant's spoken “done” is not execution evidence. See script call semantics and exposing scripts to LLMs.
Validate the two entrances before attaching a microphone
Keep the acceptance configuration separate from the household's daily configuration. Record the Home Assistant Core version, conversation agent, ASR/TTS service versions, and a checksum of the script file. Even a model-only change should rerun default rejection, an allowed action, and unavailable-state cases. Acceptance belongs to that combination, not to a component's product name. If behavior regresses, this gives the tester a specific configuration to restore rather than a recollection of what changed.
After checking and loading the package in an isolated Home Assistant instance, select the Home Assistant conversation agent and type the lab sentence. Inspect the automation trace, returned script result, and both virtual states. This fixed-sentence entrance does not need an LLM. Test both allowed and rejected branches before adding Chinese ASR and a microphone.
For Ollama, create a separate assistant configuration with a tool-capable model and enable control. Expose only the lab script through the voice-assistant settings—not the three policy helpers or unrelated scripts. Inspect the actual available tool list; an intended exposure list is not proof of what a specific integration presents. Try several bedtime phrasings and see whether they select the same script. After a rejected return, the assistant must say that no action occurred. If the model selects a different tool or ignores the result, this route has not passed; fall back to the fixed sentence rather than widening the generic execution tool.
Only then add the satellite. Verify Chinese transcription, conversation, and Chinese TTS separately. A Chinese-capable ASR says nothing about the chosen TTS voice. Judge the lab by traces, return data, and virtual entity states—not by hearing a cheerful “done.”
Where should time, occupancy and preferences live?
For an “evenings only” requirement, put a time check in the execution layer. For “dim the bedroom only when occupied,” check the occupancy entity's availability and freshness before deciding. A model can understand the phrase, but the resulting behavior needs observable states and bounded actions. Decide beforehand whether an offline sensor means skip, reject, or manual fallback; do not let the model fill in “someone is probably there.”
Brightness may be adjustable in a UI, while its hard minimum and maximum remain in configuration or an independent policy service. If you later accept “set it to 20 this time,” validate type, unit, and range at the tool boundary. A slider, field description, and system prompt are not substitutes. The lab deliberately accepts no external brightness argument so it can first teach fixed targets, configurable policy, and explicit refusal without pretending a parameterized tool has been implemented.
The script is not a transaction. If its first virtual update succeeds and the second fails, there is no automatic rollback. mode: single limits simultaneous re-entry; it is not an idempotency token across retries. With physical devices, feedback must distinguish accepted request, observed state update, partial completion, and unknown result. A verbal success claim should never exceed the evidence available from the executor.
Long-running actions need more than a single initial condition check. Home Assistant evaluates a condition against state at that point; the state can change afterward. Recheck before critical steps and add a separate stopping mechanism when necessary. Heat, mechanical, and other hazardous loads need ongoing device protection and physical measures. Replacing the virtual helper names with a thermostat does not add those protections.
“Always do that” crosses a different boundary
“Dim the night light tonight” is a one-off action. “Always dim it when I go to bed” changes future behavior. A system that learns and saves every conversational preference without a visible change step may later operate equipment for reasons nobody can explain.
A persistent change should become a readable proposal: which routine changes, old and new values, affected devices, and effective time. If bedtime brightness changes from 15 to 20, confirm that exact diff—not a vague instruction to “optimize the home.” Record a policy version, rehearse against virtual states, then let an authorized person decide whether to enable it. Reverting a policy restores an earlier configuration; it cannot undo physical actions that already happened.
Proposal, approval, versioning, and rollback are additional change-management design, not features implemented by this YAML package. A first deployment can keep the model as an explainer and have someone with configuration authority edit the rule manually. That is less autonomous, but it keeps temporary requests separate from long-term behavior and makes a bad change traceable.
Television, guests and models meet different trust boundaries
A microphone's location is not a person's identity
The last illustrative scenario is a television playing “unlock the door,” or a guest saying it to the living-room satellite. Area context tells the system where a request originated, not who authorized it. A wake word is not a password. Voice biometrics alone should not authorize a high-risk action, and a model's guess from speaking style is not identity.
Home Assistant's Acts as user option makes an assistant act for a specified account; it does not recognize the current speaker. Binding a satellite to the owner's account does not imply that a guest receives guest permissions. Different household permissions require a genuinely authenticated entry point and a verified action policy. Treat a shared-room microphone as low trust. See the Acts as user explanation.
The lab's answer for sensitive devices is not to expose them to voice at all, including through a script with a harmless-sounding name. Audit the call chain of each exposed script: what services, devices, and child scripts can it reach? Hiding a lock entity while exposing an “away mode” script that can unlock it has not removed the voice-to-lock path. Check the entity exposure list and the script's effective action reach separately.
A second spoken “yes” is not independent confirmation
The same microphone could hear confirmation from the TV, a recording, or a guest. If a sensitive action really must be available, consider requiring an authenticated phone or other trusted entrance to show the exact target and parameters for short-lived confirmation. Bind that confirmation to this action, not the whole conversation, and do not let the model substitute a new target.
An independent authorization service would need to bind a ticket to subject, action, target, parameters, expiry, and a one-time identifier, then recheck permission and state before execution. This is extra engineering design. A Home Assistant script does not gain replay resistance merely by adding a confirmed: true input. Until that layer is built and tested, keeping sensitive actions on a trusted manual path is more honest than asking the model to “be careful.”
External text, device names, or retrieved content may also contain instructions addressed to the model. A prompt can tell the model to ignore them, but the actual action boundary is the set of callable tools and their backend checks. Do not supply a general shell, arbitrary HTTP, arbitrary service calls, configuration writes, or management credentials to a home assistant. Misunderstanding a request must not expand its authority.
Inspect logs and outbound traffic before calling the system private
Even when speech and model inference happen at home, inspect whether containers, add-ons, and integrations call external services, and whether diagnostic logs retain audio, room states, or transcripts. “Local AI” describes where selected inference stages run; it does not establish that the entire system has no outbound traffic, retention, or third-party account dependency.
Expose only necessary connections among the satellite, ASR/TTS, model service, and Home Assistant; do not publish model inference ports to the internet. Separate voice-chain credentials from administrator credentials. Prefer request identifiers, result codes, needed timestamps, and redacted targets in routine logs. If sound debugging really needs recordings, obtain consent, restrict access, and set a deletion time. Agreement to voice-controlled lights is not agreement to retain every household conversation.
Test failure modes separately: internet disconnected, ASR down, LLM down, and Home Assistant down have different consequences. Deterministic text intent may not require the LLM, but a microphone still needs a functioning speech path. A wall switch may work, while a phone interface that depends on Home Assistant does not. Verify the real dependencies of a manual fallback instead of labeling a box “manual control” on a diagram.
Prove improvement with a test set, not one successful demonstration
Use fixed tasks covering same-named lights in two rooms, negation, slow pauses, TV background, the satellite's own speech, unauthorized requests, and unavailable entity states. Check target selection and policy through the text entrance first, then use recordings and live microphone input. Obtain consent and redact any real household speech; writing an article is no reason to keep a permanent family-audio archive.
When comparing two versions, hold speaker, location, noise, speaker volume, tasks, and measurement method steady. Record model and configuration versions. Replaying fixed audio isolates transcription differences but cannot reproduce a real speaker's distance, orientation, and acoustic feedback. Keep tuning utterances separate from acceptance utterances so the result is not merely good on phrases the team has already seen.
| Stage | Record | What it does not prove |
|---|---|---|
| Wake | False activations over a fixed period; misses among intended activations | Correct transcription or control |
| Chinese transcription | CER after a consistent normalization; errors on negation, device and room names | A low CER does not prove the correct target |
| Intent and target | Correct action, entity and parameter when the transcript is correct | A plausible spoken reply is not enough |
| Execution | Confirmed, rejected, wrong, partial and unknown outcomes separately | Sending a request is not device completion |
| Response | End of speech to first feedback and to state confirmation, p50/p95 with sample counts | One best-case run is not daily experience |
| Degradation | Which entrances work or reject when each service is unavailable | Internet loss is not the only failure |
CER can conceal a dangerous error: losing one negation in a long sentence may leave a low character error rate while reversing the action. Conversely, a mangled room name that leads to clarification is an unfinished request, not a wrong action. Keep safety refusals separate from errors; combining them into one “failure rate” can tempt a team to remove necessary checks in pursuit of a prettier success number.
The virtual lab should eventually be tested for default rejection; updates to both virtual states only after explicit enablement and leaving guest restriction; renewed rejection when the restriction returns; bounded brightness; and rejection when a required state is unavailable. These are expected tests to run in an isolated Home Assistant instance, not results we have obtained. Inject unavailable states only in a controlled lab, not by creating hazardous conditions on physical devices.
After a wrong action, disable the script's voice exposure and the related sentence automation, retain necessary traces, and return to text or trusted manual control. Keep the last verified configuration; do not change model, aliases, and action script at once. After a partially completed physical routine, read current states before considering compensation. Never blindly replay the whole routine. Locks, heat, gas, and life-safety equipment are not suitable targets for this voice-tuning exercise.
A local voice system is worth expanding when it can answer four specific questions: What words were transcribed? Why was this device selected? Which rule permitted the action? What final state was actually confirmed? If any answer is missing, improve that boundary and its observability before adding freer conversation. The result is not merely a more talkative remote control; it is a control path that can stop at ambiguity, adjust bounded parameters, and show who or what permitted each action.
For the broader placement decision, see Home Assistant voice: local, cloud or hybrid. Turning a voice endpoint, model service, and device controller into a product also requires interface, authorization, interaction, and failure-mode design together—not simply a working model API.
