Kyutai's Moshi model can listen and speak at the same time

What happened

On 3 July 2024, the French research lab Kyutai streamed a live public unveiling of Moshi, describing it as 'the first voice-enabled AI openly accessible to all.' The underlying research paper followed, first posted on 17 September 2024 and revised in October, alongside open weights and code on GitHub. The gap between the demonstration and the paper's posting is typical of how speech-model labs work: a public demo first, a full technical account afterward.

What the documents show

The paper's abstract calls Moshi 'the first real-time full-duplex spoken large language model' and reports a 'theoretical latency of 160ms, 200ms in practice.' The GitHub repository breaks the theoretical figure down further, as '80ms for the frame size of Mimi plus 80ms of acoustic delay,' with the practical figure measured 'on an L4 GPU.' Both documents agree the model treats a user's speech and its own output as two parallel streams rather than a single alternating channel, which is what lets it be interrupted mid-sentence or overlap with a speaker instead of waiting for a detected pause. The code carries an MIT or Apache license depending on component, and the trained weights are released under CC-BY 4.0.

The mechanism

'Full-duplex' names a specific engineering change, not a vague claim of naturalness: most voice assistants process a caller's turn, decide the turn has ended, then generate and speak a full reply, a strictly alternating pipeline. Moshi instead models both streams continuously, so it can vocalize a brief acknowledgement, be talked over, or resume speaking without the discrete turn-taking that produces the slight pause callers usually hear from a companion app. Latency in this design is not one number but two: the theoretical floor set by how much audio the model must buffer before it can act, and the practical figure measured on specific hardware, which will vary with the server a company actually runs.

What it leaves open

The cited documents describe Moshi as a research release rather than a finished companion product, and neither states how the model behaves on hardware slower than the L4 GPU cited, nor how full-duplex turn-taking performs in a long, emotionally loaded conversation rather than a benchmark exchange. Kyutai's own site separately claims the model 'can understand emotions,' a broader claim than the paper's latency and turn-taking results establish on their own.

• Is a quoted latency figure theoretical, or measured on the hardware a given companion app actually runs?
• Does 'full-duplex' in a product description mean continuous parallel streams, or just a shorter pause between turns?
• What happens to interruption handling and turn-taking in a long conversation, beyond a short demonstration?

Moshi is evidence that full-duplex, low-latency spoken dialogue is achievable and openly documented, not evidence about how any particular commercial companion app built on similar ideas will actually perform for a user.