All the tooling around them not keeping up anyway.
Actual “AGI” being distant and completely unrelated to contemporary models.
Inference costs plummeting.
The last one is critical.
Right this second, you can run Minimax H3 on a desktop for the tiny fraction of the compute/RAM OpenAI Sora took. And it’s better.
In a month, it will be ~8X faster.
You can run DeepseekV4 flash, dirt cheap, and get what Claude was less than a year ago. And it’s gonna spread to every host out there, to systems like Cerebas ASICs that don’t even need HBM.
So… Even if you’re an AI acolyte. And we go with that for the sake of argument…
We don’t actually need all that RAM for hosting generative models?
I’m very interested to see what happens to all these datacenters over the next two-three years.
They spent all this money on something that’s gonna be cheap as dirt to run, largely run locally, and that won’t need GPUs once bitnet takes off, sooo… they can’t make money off that.
What happens then?
What happens to all those Stargate RAM wafers, and excess datacenters?
Lol what? I just checked, on eBay I can find a lot of offers between 800 Euros and 1300 Euros for a used RTX 3090.
4-5k would get you a brand new RTX 5090.
Okay, not as bad as I thought, there’s a guy selling a 4090 for “only” 2k. There’s nobody selling a 3090 in the local hardware forum. I’m not going to buy off ebay because if someone on ebay sells a dud, I can’t go back to punch him in the face.
5090s start at about 5200, that’s true. To be fair, the “4-5k” for a used 3090 was a bit hyperbolic, I’m just mad that they’re so expensive that my last 2 cars cost less than a USED 4090.
You are 100% correct IMO for the AI stuff, just a question, what’s bitnet? Is it some supposed bitcoin-killer (because that’s not going to happen, IMO).
I also think we’re on the cusp of “having enough” personal computing power with the usual exceptions, take a bit, give a bit.
Our mobile phones are close to what’s needed for most people (8GB, 4k, gaming, …). Or so I think!
Bitnet is a catch-all title for ML models that use a specific mathematical trick.
If elements of a matrix are composed of only 1, 0, or -1, multiplying them is the same as as adding them.
That’s huge. LLM computation is basically all matrix multiplication, so if you replace that with simple addition, you reduce the computational requirements by orders of magnitude.
The catch is such models are hard to train effectively; its proven that it works, but research to get the technique usable and practical is still being done.
Personally, I suspect it’s unviable for many “dense” parts of models, but sparse hybrid bitnet models would be really cool.
I mention it because, if it takes off, suddenly the massive matrix multiplier accelerators we have for LLMs aren’t as useful. Chips with simpler architectures could get the job done, at least for parts of models that are bitnet.
There are indeed some Bitnet LUT implementations, already.
It’s both smaller and faster though; the models are drastically smaller than FP16 at the same parameter count, as they’re just 1.5 bits instead of 16.
How much is lost from the precision drop is an open question, though, and the answer seems to be complex and finicky. I think it’s why big labs are hesitant to pursue it; they want to try safer, lower hanging fruit first. Also, it’s not particularly efficient on hardware (like AI GPUs) where most of the silicon is dedicated to matrix multiply performance anyway.
Well, the current trajectory points to:
LLM capabilities topping out.
All the tooling around them not keeping up anyway.
Actual “AGI” being distant and completely unrelated to contemporary models.
Inference costs plummeting.
The last one is critical.
Right this second, you can run Minimax H3 on a desktop for the tiny fraction of the compute/RAM OpenAI Sora took. And it’s better.
In a month, it will be ~8X faster.
You can run DeepseekV4 flash, dirt cheap, and get what Claude was less than a year ago. And it’s gonna spread to every host out there, to systems like Cerebas ASICs that don’t even need HBM.
So… Even if you’re an AI acolyte. And we go with that for the sake of argument…
We don’t actually need all that RAM for hosting generative models?
I’m very interested to see what happens to all these datacenters over the next two-three years.
They spent all this money on something that’s gonna be cheap as dirt to run, largely run locally, and that won’t need GPUs once bitnet takes off, sooo… they can’t make money off that.
What happens then?
What happens to all those Stargate RAM wafers, and excess datacenters?
Deepseek v4 Flash is insanely good and basically free. Especially the latest model which I think is only available from China right now.
Nah, 0731 weights were released. I’m running it locally right this second.
I don’t know about Deepseek API specifically, but there are tons of places to get it.
You are Mr. moneybags or you’re running 2 or 4bit quantized.
I have a 3090, 7800, and 128GB ram from pre-rampocalypse. It’s not nothing, but there are definitely crazier home labs.
Deepseek V4 is native 4 bit.
I am running this quant, but the quantization loss is measurably low: https://huggingface.co/Downtown-Case/DeepSeek-V4-Flash-0731-128GB-RAM-IK-GGUF
The native MXFP4 is only a bit bigger.
A used 3090 alone is like what, 4-5k now?
Your setup isn’t really achievable for most now.
Lol what? I just checked, on eBay I can find a lot of offers between 800 Euros and 1300 Euros for a used RTX 3090. 4-5k would get you a brand new RTX 5090.
Okay, not as bad as I thought, there’s a guy selling a 4090 for “only” 2k. There’s nobody selling a 3090 in the local hardware forum. I’m not going to buy off ebay because if someone on ebay sells a dud, I can’t go back to punch him in the face.
5090s start at about 5200, that’s true. To be fair, the “4-5k” for a used 3090 was a bit hyperbolic, I’m just mad that they’re so expensive that my last 2 cars cost less than a USED 4090.
The 3090 is from 2020. It’s under $1300 based on eBay prices I’m seeing (which is around what it launched at).
Crazy expensive, but not $5K.
The 128GB RAM is madly expensive now, though.
You are 100% correct IMO for the AI stuff, just a question, what’s bitnet? Is it some supposed bitcoin-killer (because that’s not going to happen, IMO).
I also think we’re on the cusp of “having enough” personal computing power with the usual exceptions, take a bit, give a bit.
Our mobile phones are close to what’s needed for most people (8GB, 4k, gaming, …). Or so I think!
Bitnet is a catch-all title for ML models that use a specific mathematical trick.
If elements of a matrix are composed of only 1, 0, or -1, multiplying them is the same as as adding them.
That’s huge. LLM computation is basically all matrix multiplication, so if you replace that with simple addition, you reduce the computational requirements by orders of magnitude.
The catch is such models are hard to train effectively; its proven that it works, but research to get the technique usable and practical is still being done.
Personally, I suspect it’s unviable for many “dense” parts of models, but sparse hybrid bitnet models would be really cool.
I mention it because, if it takes off, suddenly the massive matrix multiplier accelerators we have for LLMs aren’t as useful. Chips with simpler architectures could get the job done, at least for parts of models that are bitnet.
Thank you! Ah yes that makes sense somewhat I guess. Trading cpu usage for memory usage, an old classic 😁!
Nibble * nibble look up tables could probably be fast too, it all depends on where you want the complexity I guess.
There are indeed some Bitnet LUT implementations, already.
It’s both smaller and faster though; the models are drastically smaller than FP16 at the same parameter count, as they’re just 1.5 bits instead of 16.
How much is lost from the precision drop is an open question, though, and the answer seems to be complex and finicky. I think it’s why big labs are hesitant to pursue it; they want to try safer, lower hanging fruit first. Also, it’s not particularly efficient on hardware (like AI GPUs) where most of the silicon is dedicated to matrix multiply performance anyway.
Lower inference costs will lead to more demand for RAM
Will it though?
I posit there’s a saturation point where there’s “enough” LLM in use, and we are not far from that.
The whole pitch from Altman and such is scaling models up. But that’s not working.