Latency budget: where do the milliseconds go in a voice loop?

Capture, VAD, ASR, LLM, TTS, playback. Share your numbers.

Subscribed to this thread 🔔

Does the latency budget work offline, or does it need the cloud?

Code snippet, should work with minimal changes:

async def stream(frames):
    async for chunk in frames:
        yield vad(chunk)

Benchmarked on my machine: ~220 ms to first token with a quantised 7B. ✨

@hannah-t Code snippet, should work with minimal changes:

async def stream(frames):
    async for chunk in frames:
        yield vad(chunk)

@Nadia_S67 One more vote for the latency budget. Please keep improving it! 😅

Is it just me, or did the latency budget change after the last update?

I'd love to see a short demo of the latency budget. Has anyone recorded one?

Good questions. Answering one by one below.