Running Qwen and Gemma locally: quantisation trade-offs

Q4 vs Q5 vs Q8: what did you pick and why?

mert-k, vgarcia, turner_derek and 7 others like this.

Code snippet, should work with minimal changes:

async def stream(frames):
    async for chunk in frames:
        yield vad(chunk)

rnair and mariam-n like this.

Appreciate the honest feedback, even the harsh bits 😄

cnwosu, wanjiruW, Thiago_C25 and 13 others like this.

161

We moved to streaming everything and p95 dropped a lot. 😅

@Zainab_Sheikh We moved to streaming everything and p95 dropped a lot.

ssato, imran-k, marcusT and 8 others like this.

112

Does quantisation work offline, or does it need the cloud? 😅

@krueger_moritz We discussed quantisation in our team chat today and everyone had a different opinion 😅