Voice AI Executives Say Context Still Limits Natural Conversations

Voice AI executives at PolyAI and Otter said speech recognition and limited context remain major barriers to natural conversations and reliable follow-up actions.

Shawn Wen, chief technology officer at PolyAI, said full-duplex models have reached an important milestone. These systems can listen and speak at the same time, but the next challenge is making reasoning fast enough to fetch answers quickly and keep conversations natural.

Customer-service AI agents also need to sound less robotic and give callers confidence that they can resolve problems, Wen said. That confidence may develop after an agent successfully handles the first two or three turns of a conversation, he added.

Alex Gay, chief marketing officer at Otter, said meeting automation depends on identifying speakers, capturing intent and connecting transcripts with organizational knowledge. Otter is also developing digital twins that could represent people in meetings.

Gay said those systems need voices that convey human-like emotional expression during debate and strategic discussion. Without that quality, an avatar may function more like a question-and-answer chatbot than a participant in a relationship-based conversation.

Speech recognition remains a central weakness. Wen said automatic speech recognition models can miss important keywords, making it harder to preserve the full context of a conversation.

Gay said transcription is the foundation for Otter’s productivity tools rather than the final product. When the initial transcript is inaccurate, summaries and follow-up actions can also be flawed, undermining trust in the platform.

Voice AI companies also face transparency questions about recording and disclosure. Otter is exploring methods such as notifying everyone in a meeting chat when a conversation is being recorded, including meetings in which its bot is not present. Wen said enterprise users should also be told when they are speaking with an AI system.