The speaker is halfway through an answer when someone says, “Wait a moment.” Playback stops, but recognition receives only “a moment.” In another version of the same frustration, the device keeps talking and treats its own answer as a new question. A useful interruption has to preserve the person’s voice inside the acoustic mixture and retire the old audio already waiting to play.
Follow that short phrase through the buffers. The discussion below uses pinned ESP-SR source interfaces on ESP32-S3 as a design reference. It is not a flashed-board report: there are no synchronized recordings, measured interruption times or success rates. The point is to locate where the first word can disappear, rather than keep adjusting a single sensitivity setting.
Two voices arrive at one microphone
Listen to the raw microphone channel first. The speaker’s answer travels through the enclosure, desk and room before returning to the microphone. When the person speaks over it, the recording contains both near-end speech and playback echo. Muting capture throughout playback removes the device’s own voice, but it closes the very entrance needed for spontaneous interruption.
Acoustic echo cancellation needs a second piece of information: the sound being sent toward the speaker. That playback reference helps estimate the echo component in the microphone signal while retaining near-end speech. The reference describes what the device is saying; the microphone captures what the room actually delivers. A loudspeaker and an acoustic path sit between them, so subtracting identical digital samples does not model the whole transformation.
The pinned ESP-SR AFE guide identifies microphone channels as M, playback reference as R and unused channels as N. AFE input is channel-interleaved. Match peripheral slots to those roles before feeding the buffer. A program can process correctly sized buffers indefinitely while using an empty slot as its reference; memory size alone will reveal nothing about that mistake.
An all-zero R channel deserves different interpretations depending on playback. During audible output, follow the reference wiring and buffer population. With the speaker genuinely silent, zeros can be expected, and the useful question becomes whether near-end speech survives the processing chain. Issue 235 asks whether zero reference means raw microphone output, but supplies no answer that establishes that guarantee. Other processing stages and the selected configuration still matter.

Generated illustrative assembly, not a specific development board, wiring guide or tested product. Verify placement and reference recovery on the target hardware.
Give the canceller the sound that actually played
A tempting reference source is a copy of newly received network audio. At that point it may still be compressed and must pass through decoding, resampling, volume adjustment and queues. The microphone hears something played later. Let the network receiver run far ahead and the reference can run ahead with it, separating two signals that the canceller needs to relate.
Espressif’s microphone design guide recommends a reference close to the speaker side and discusses recovery after DA and before PA. On a particular board, identify the actual tap on the schematic: digital playback copy or analog recovery, the gain and distortion it includes, and whether it shares a sampling timebase with capture. The answer depends on the codec and amplifier rather than a universal wiring shortcut.
Short synchronized microphone and reference recordings are useful at playback start, mute and volume changes. A changed speaker volume with an unchanged reference suggests investigating gain placement. An offset that keeps growing suggests clocks, resampling or missing frames. A fixed delay and a drifting offset need different repairs; moving one buffer once cannot correct a timebase that continues to diverge.
Check amplitude while looking at that relationship. Excessive output can clip microphone capture or the recovered reference ADC, changing the waveform before software sees it. Increasing suppression may then weaken the person’s speech as well. Gain, acoustic cavities, pickup openings and mechanical isolation belong in the investigation. The official guide also discusses separating microphones from speakers and vibration sources. Parameters that work on an exposed board need another assessment inside a changed enclosure.
For the 16 kHz AFE path discussed here, query the actual feed chunk size and channel count, and provide signed 16-bit samples. Convert at the boundary if the network service uses another rate. An illustrative time slice in an article should never become a hard-coded library frame size. The separate AEC API also has its own buffer layout; AFE interleaving is not a universal DSP convention.
Take an M, N, R slot arrangement. Each sampling instant must keep those three slots together. Concatenating three mono arrays instead would make adjacent microphone values look like different channels, even if the total byte count were correct. Recognizable offline input can help check packing before acoustic tuning starts. That is a proposed development technique, not a test performed for this article.
A reference can also be present but stale. If a playback producer blocks while capture repeatedly reuses its previous reference block, both lengths can remain plausible while content freezes. Advancing sample positions for capture and reference updates exposes more than a “task alive” message. Record gaps explicitly. Whether a gap requires silence insertion or processing-state recovery depends on the selected implementation; silently repeating old samples hides the problem.
Local notification sounds belong in this picture too. A reference covering only remote TTS misses a connection chime mixed into the same speaker output. Identify all sources in the final mix and whether the tap is before or after mixing. Multiple outputs need the same attention to actual echo paths. Otherwise interruption works during an answer but triggers incorrectly during a chime, making two different reference routes look like an intermittent algorithm failure.
Keep the person audible during double-talk
When only the speaker talks, a quiet processed output is easy to praise. Add a person speaking over it and the task becomes more demanding: retain intelligible near-end syllables while reducing playback echo. If remote-only output is clean but the person fades during double-talk, compare raw microphone, reference and processed audio together. Listening only to the silent sections rewards the wrong behavior.
The pinned ESP32-S3 AEC header distinguishes SR, VOIP and FD modes, including low-cost and high-performance FD choices. The AFE configuration header exposes a full-duplex type as well. An introductory SR example helps establish the API, but a talking-and-listening product still needs its actual mode, models and resource configuration checked.
The AEC header also exposes nonlinear post-processing levels and warns of the desired-audio quality tradeoff with stronger suppression. Listen for the interruption’s first consonant, broken vowels or speech that disappears after a volume change. An FD enum is an available configuration, while the result depends on the reference, enclosure and processing chain around it.
Read the contract of the entry point actually called. In this pinned revision, the esp_afe_aec.h wrapper says it selects the first microphone and reference channels. A broader description of dual-microphone support does not establish that this particular wrapper processes every supplied microphone. This distinction is especially useful when moving between an AFE example and a lower-level AEC integration.
Use the same near-end phrase in quiet, playback-only and overlapping-speech material. If quiet near-end speech is intact and overlap damages it, investigate echo processing and its post-processing interaction. If all material sounds poor, capture gain, microphone placement or input formatting deserves attention first. A nearly silent output can have little residual echo and still be useless for conversation.
An echo attenuation calculation also needs a suitable interval, such as far-end-only playback. During double-talk, part of the output should be the retained person’s voice. Treating all that energy as unwanted echo would penalize the desired result. Intelligibility and downstream recognition add information that a single energy ratio cannot capture. No decibel improvement is reported here because the required recordings do not exist for this design.
Offline replay is useful for comparing processing configurations against the same input. A physical speaker and enclosure are still needed to assess the actual acoustic loop. Clean speech played through headphones cannot stand in for microphone input at maximum device volume. Separate data-handling checks from real volume, distance and orientation checks, and label their evidence separately.
If continuous capture, processing, encoding and playback exceed the product’s resources, a button, a wake-word interruption or half-duplex operation can be an honest alternative. The user pays with an explicit interaction step or waiting time. Merely running a capture task beside a playback task leaves the acoustic work unresolved.
That interaction choice changes the scope of ESP32 firmware development. Half-duplex can organize work around switching between capture and playback; spontaneous interruption adds concurrent capture, reference, detection and output handling. Agreeing how the person interrupts during requirements work makes the change surface clearer than adding a full-duplex checkbox after a basic conversation already works.

Concept only, not a recording or performance plot. Blue represents playback and orange near-end speech. The illustration explains what to preserve, without promising the depicted suppression.
The first word precedes the VAD decision
Suppose processed audio now contains a clear “Wait a moment,” while the recognition request still starts at “a moment.” Look at voice-activity detection next. The system may decide that speech has begun only after initial sound has already crossed the processing chain, then start uploading. The missing word was available but never included in that request.
The pinned VADNet documentation describes onset delay from the algorithm and continuous-trigger requirements, and provides a VAD cache to retain earlier audio. When fetch returns a cache, prepend it to the current speech for the same upload. Use the returned size rather than substituting a presumed default duration.
For arithmetic only, imagine an application that starts saving audio 100 ms late. At 16 kHz that interval contains 1,600 samples, or 3,200 bytes of mono 16-bit PCM. This converts time into a buffer requirement; 100 ms is neither a measured delay on this board nor a recommended VAD setting. It helps answer how much of a spoken onset a proposed pre-roll can cover.
Longer pre-roll costs memory and upload volume and can carry material from a previous turn. Too little continues to clip speech. Associate buffered samples with capture time and determine the exact join to live audio. Overlapping the cached and live regions can upload the first word twice, creating a different recognition symptom from losing it.
Asynchronous upload adds an ownership question. A processing task can fetch another result while the network worker still holds an older pointer. Do not assume returned storage remains unchanged without an interface lifetime guarantee. Copy into application-owned bounded storage or use an ownership-transfer mechanism explicitly supported by the API. Copying consumes memory and work; borrowing requires a contract that actually permits it.
VAD indicates speech activity rather than the identity of a consenting user. Television, another person or residual playback can influence it. Free interruption, wake-word interruption and a button are distinct product choices. Set the intended interaction before using detection to trigger it. Lowering a threshold changes both missed speech and unwanted triggers.
Disabling capture can hide false triggers along with genuine interruptions. A more informative diagnostic is a short window around the trigger, correlated with playback state: does it contain near-end speech, echo or a local chime? User recordings require authorization and bounded retention. Sample positions and redacted event times can support longer-lived diagnostics without retaining every raw conversation.
Retire the answer still waiting to play
The phrase has reached the next request intact, yet the old answer continues from the speaker. There may be unfinished generation at the service, data in transit, decoded PCM in device queues and samples already handed to peripheral DMA. Each layer owns a small piece of the past response.
Consider a hypothetical queue containing 4,800 mono samples at 16 kHz. It represents 300 ms of playback, or 9,600 bytes of 16-bit PCM, even if the server sends nothing more. A real device may use another rate and has additional peripheral and scheduling behavior. The arithmetic explains why sending cancellation can precede audible silence without establishing a measured end-to-end delay.
A possible handoff design first prevents the old turn from entering the player and stops or flushes queued output according to peripheral capabilities, while requesting the service’s supported cancellation. Preserve input pre-roll for the new request and admit the new answer into a new playback generation. Application-owned blocks can carry a local generation tag so the player rejects retired work. That tag is a design proposal, not a field promised by every voice protocol.
The pinned Xiaozhi WebSocket protocol describes an abort message and session identity. It does not establish a cancellation acknowledgement or a turn identifier on every binary audio frame. If late audio cannot be reliably attributed, agree a stronger client/server contract, reopen at a defined protocol boundary or use another verifiable isolation mechanism. Relabeling an unidentified late packet locally cannot reveal which remote generation produced it.
Reference handling must follow the actual output as it stops. Continuing to feed discarded old speech after the speaker is silent breaks the relationship established earlier. Clearing R while DMA is still sounding also leaves an uncovered echo interval. Coordinate output progress and reference updates using the actual codec, DMA and recovery arrangement.
Clearing a queue also races with its producer. A decoder holding an old block can enqueue it immediately after a flush. Make playback-generation changes and admission checks part of one controlled transition so retired blocks lose eligibility. Avoid holding a high-priority audio task waiting for a network cancellation acknowledgement; local output control can advance while service state is resolved separately.
Place raw microphone audio, actual reference, processed near-end audio and detection/playback events on one timeline. If the microphone contains “Wait” but processing does not, investigate reference and AEC. If processing retains it but upload does not, inspect VAD cache and joining. If the new request is complete but old speech remains audible, inspect output queues and cancellation. Keep playback content, volume and speaking position comparable when changing one factor. This is a localization method, with no product test result claimed.
The aim is a complete interruption: the first word survives while the device speaks, reaches the new request with its onset, and causes the previous answer to become quiet. The model then has the words needed to answer the person’s changed question.
For the broader division between endpoint, model and product backend, continue with where a spoken turn goes in an ESP32 voice terminal. This article stays inside that journey long enough to keep one interruption intact.
