Brett@programming.devtoTechnology@lemmy.world•Generative AI Is an Engineering Disaster. A shockingly inefficient trillion-dollar project.English
4·
28 days agoIs that quantized? 4 bit Qwen 3.6 can get 22tps on a 1060.
C++ Programmer
- 0 Posts
- 2 Comments
Joined 1 year ago
Cake day: March 9th, 2025
You are not logged in. If you use a Fediverse account that is able to follow users, you can follow this user.
What are you using to run the model? Llama.cpp will automatically split the model between your system ram and graphics card’s vram.
Qwen 3.6 is a mixture of experts model with only 3B parameters active at a time. Even without quantization your card could easily run that.