To use one equivalent to Claude Opus, you need like 800GB GPU memory. The chips that will get you there run $20-30k a pop, and you’d need 4 or more of them.
That’s the biggest baddest model out there. There are models that get you 90% of the way there with a significantly smaller parameter count and thanks to MoE offloading you don’t need the entire model active at once.
A 5090 and a beefy CPU and tons of RAM won’t be cheap or even affordable to most, but you could run very big models and have a beefy PC for other activities. But even a 16 GB card could do plenty.
To use one equivalent to Claude Opus, you need like 800GB GPU memory. The chips that will get you there run $20-30k a pop, and you’d need 4 or more of them.
That’s the biggest baddest model out there. There are models that get you 90% of the way there with a significantly smaller parameter count and thanks to MoE offloading you don’t need the entire model active at once.
A 5090 and a beefy CPU and tons of RAM won’t be cheap or even affordable to most, but you could run very big models and have a beefy PC for other activities. But even a 16 GB card could do plenty.
Yeah, I’m aware. I have realistic expectations and I’m looking into running something simpler and less demanding.