A microphone captures a question and a speaker answers it. That does not mean the intelligence between them runs on the ESP32. For a custom voice terminal, a useful division is to keep sound and immediate interaction on the device, open-ended understanding and generation in a model service, and admission and business permissions in the product backend. Direct media connectivity to a model does not make the backend's responsibilities disappear.
Following one spoken turn exposes those boundaries more clearly than choosing a model name first. Distorted input cannot reliably be repaired by later text reasoning. A generated answer can still be waiting in a playback queue. Without distinct responsibilities, firmware and backend dashboards may both show success while the person in front of the terminal is still waiting.
Before the network, there is an audio product
Use a concrete reference: an ESP32-S3-Korvo-2 V3.1 audio board connected over Wi-Fi, without adding a display, camera or person-to-person calling to the design. The basis is the official board guide and Espressif's WebRTC example. This is a review of public designs, not a report of a board flashed and tested for this article.
The board uses an ES7210 ADC for capture, an ES8311 codec on the playback side, and a defined echo-reference path. Firmware must distinguish microphone samples from the reference associated with playback. A smaller board with a digital microphone may use the same division of responsibilities, but it cannot inherit this channel configuration unchanged.
The ESP-SR audio front end identifies microphone, playback-reference and unused channels as M, R and N, with interleaved input. Its feed interface uses signed 16-bit, 16 kHz audio. That is an AFE input contract, not a declaration that every cloud API or WebRTC track must receive those same bytes. Conversion belongs at the boundary where formats change.
Echo handling, noise processing and voice-activity detection prepare or segment sound; wake-word detection decides when interaction should begin. None of those jobs is open-ended language understanding. A terminal lighting up after a wake word can therefore still lose the rest of the question. A wake indication is evidence of one component, not acceptance of the complete path.
Immediate interaction also belongs on the endpoint. Listening, connecting, muted and offline need distinguishable feedback. A physical or local mute action should not depend on the remote model first understanding a request to stop listening. The device must enforce that promise and expose its state.

Generated generic hardware scene, not a Korvo-2 photograph, wiring diagram or test result.
Two journeys beyond the microphone
A transcript between speech and reasoning
In a chained design, automatic speech recognition turns input into text, a language model and business logic process that text, and text-to-speech converts the answer into audio. The ESP32 can remain an audio endpoint while those services run on a local server or a hosted platform. A locally deployed service is not necessarily inference inside the microcontroller.
Intermediate text makes some failures easier to locate. Inspect transcription when a name is wrong; inspect resource mapping when the words are right but the wrong device is selected; inspect synthesis and rendering when the written answer is correct but the sound is not. Separate components can be replaced independently. The price is ownership of fragment ordering, turn correlation and error propagation across those components.
A chain does not require recording a complete utterance before uploading it. Streaming is possible, but submission still needs a policy. Committing too early can split one request into two; waiting for a complete sentence changes the pace. Choose that boundary around the product task rather than treating one demo's recording behavior as an architectural rule.
Audio into a realtime conversation service
A realtime voice service can receive and generate audio without the application explicitly assembling three independent ASR, language-model and TTS services. The endpoint exchanges media and also manages session and tool events. Removing an application-managed chain does not remove state: the product must still know whether the session exists, which answer is current, and whether the user has left.
Espressif's pinned ESP-WebRTC example combines capture, rendering, a WebRTC connection and function-call handling. Its Opus media configuration and custom signaling demonstrate that compressed network audio, internal PCM and conversation events are different objects. An array suitable for an I2S renderer is not automatically valid input to an arbitrary model endpoint.
This route reduces some application-level assembly work while increasing dependence on the provider's session and event semantics. Available transcription should not automatically be treated as the sole execution record, perfectly synchronized with every sound. A chain remains useful when independent recognition replacement or transcript editing is central. A realtime path is worth evaluating for conversational exchange, but the example alone does not establish lower end-to-end latency.
Solid edges show audio or text processing; dotted edges show admission and tool proposals. The two paths are alternatives, not a requirement to run both. The product backend need not relay all audio.
A connection is smaller than a conversation
A transport connection carries audio and events. A product session also describes who is interacting with which terminal. Several turns can share one connection, while reconnecting can create a new provider session. A device MAC address alone cannot distinguish events before and after a firmware restart or explain why an old answer arrived during a new request.
One design convention is to track device, session, turn and response identities separately, with explicit invalidation rules. These are suggested product fields, not universal vendor protocol names. The endpoint should be able to reject playback belonging to an expired turn even if the connection delivering it is still open.
Stopping generation and stopping playback are separate operations. When the user presses stop, firmware can silence local rendering and then request cancellation through the selected service interface. Audio already received into a queue does not disappear because generation ended remotely. Detailed double-talk and barge-in algorithms deserve their own treatment; the responsibility split is necessary before implementing them.
Error feedback follows the same principle. Failed admission should not leave the terminal indefinitely “thinking.” Missing input and an unavailable answer should not collapse into the same generic network message. Keeping failures attached to their boundary lets the product choose between reconnecting, asking for another utterance, or ending the turn.
The backend may be outside the media path
“The ESP32 connects directly to the model” can describe where media travels. It should not imply distributing a long-lived service key to every shipped terminal. The current OpenAI WebRTC documentation describes server-issued short-lived client credentials. A hardware product using this pattern still needs device admission, issuance limits and a way to revoke access for a lost or retired device.
The distinction from the Espressif demo matters. That example uses a configured API key to request a temporary credential before establishing its session. The presence of a temporary token does not establish that the long-lived key has left the endpoint. A product must relocate issuance or use an appropriate backend-established session flow rather than merely replacing the demonstration's key with a company-wide credential.
The backend has another job when the model proposes a function call. A valid function name and parseable arguments establish syntax, not permission. Business execution must check the identity, permitted resources, parameter boundaries and actual outcome. A spoken claim of completion is not a substitute for a returned device or business result.
Some terminals need no execution tools at all. A question-answering product can omit them and keep a smaller permission surface. Add account lookup or equipment changes only when those tasks are part of the product. Extending a demonstration light-control function into unrestricted administration creates a very different system.
Measure the waits before upgrading a component
Perceived delay includes capture, deciding when to submit, transport, service processing, buffering and physical playback. Changing the model affects only part of that path. Receiving the first returned audio packet does not mean the speaker has produced the first audible syllable. Record those moments separately instead of using server logs as a proxy for the entire experience.
Larger playback queues can absorb some arrival jitter but leave more old sound waiting when a user stops the answer. Buffers and codecs also share resources with TLS, networking and audio tasks. Target-firmware measurements must establish concurrent headroom; the module's advertised PSRAM capacity is not enough to prove the workload fits.
Operating cost extends beyond the price of an answer. Credential issuance, concurrent sessions, audio transfer, retention and support all belong to the product budget. Record redacted errors and timing where they suffice. Keeping every recording indefinitely adds privacy exposure rather than automatically making a system easier to diagnose.
When that complexity exceeds the task, reduce the promise. For a bounded set of commands, ESP-SR MultiNet offers an offline command-recognition route. That is not the same product as unrestricted offline conversation.
What remains when connectivity disappears?
A network-dependent terminal should still be able to mute, stop rendering and indicate that it is offline. Whether it can also recognize commands depends on the local model and mappings actually included in the firmware. An ESP32-S3 on the bill of materials does not by itself grant an offline version of every advertised conversational feature.
A useful first delivery boundary is one closed turn: accept authorized input, receive a correctly associated answer, render it, and finish clearly. Then test expired credentials, an unresponsive service, audio arriving after exit, and recovery from disconnection. These are proposed validation scenarios, not tests completed for this article, and they do not establish a person-to-person calling system.
This explanation uses the Korvo-2 V3.1 reference, a pinned ESP-WebRTC source commit and public interfaces checked on 19 November 2026. ESP-IDF 5.4 is a README entry point, not a dependency lock verified here. A real build still needs frozen board configuration, component versions and service parameters, followed by compilation, acoustic, network and sustained-operation testing before latency or reliability commitments.
For the preceding chip-level decision, see what each ESP32 family does in a product. For a custom product, ESP32 development can frame the firmware and backend scope. Assign the sound, session and permission boundaries to real owners; “connect it to AI” is not yet that scope.
