To use one equivalent to Claude Opus, you need like 800GB GPU memory. The chips that will get you there run $20-30k a pop, and you’d need 4 or more of them.
That’s the biggest baddest model out there. There are models that get you 90% of the way there with a significantly smaller parameter count and thanks to MoE offloading you don’t need the entire model active at once.
A 5090 and a beefy CPU and tons of RAM won’t be cheap or even affordable to most, but you could run very big models and have a beefy PC for other activities. But even a 16 GB card could do plenty.
Where I work, Chinese models are banned due to legal concerns. Not just in production, on any company-owned machine. That basically eliminates all the decent open weight models. I’m imagining this type of policy will be more widespread. I suppose it’s because of the potential for legal woes if systems and people are dependent on these models and then federal or state laws impose harsh penalties with little time to react.
Local AI it is then. Not that I’m using all that much now anyway…
To use one equivalent to Claude Opus, you need like 800GB GPU memory. The chips that will get you there run $20-30k a pop, and you’d need 4 or more of them.
That’s the biggest baddest model out there. There are models that get you 90% of the way there with a significantly smaller parameter count and thanks to MoE offloading you don’t need the entire model active at once.
A 5090 and a beefy CPU and tons of RAM won’t be cheap or even affordable to most, but you could run very big models and have a beefy PC for other activities. But even a 16 GB card could do plenty.
Yeah, I’m aware. I have realistic expectations and I’m looking into running something simpler and less demanding.
Where I work, Chinese models are banned due to legal concerns. Not just in production, on any company-owned machine. That basically eliminates all the decent open weight models. I’m imagining this type of policy will be more widespread. I suppose it’s because of the potential for legal woes if systems and people are dependent on these models and then federal or state laws impose harsh penalties with little time to react.
I meant for private use.
As for work, we do use AI quite a lot and I don’t have a say over what’s available.
Copilot? Copilot.