Does anyone run their own AI service?
If so,
Which one?
What hardware?
Do you have a way to access it from outside or from a phone?
And, if you’re open to sharing, what do you do with it?
I know AI-enabling hardware is insanely expensive right now, but some of the smaller (<32GB) more efficient models (think deepseek- or qwen-based) don’t absolutely need specialized hardware to run and perform well enough for many purposes.
I am so aware of the absolutely abysmal privacy practices of basically all AI companies that I’m not comfortable using them to help me with web searches and the such, because there’s no way that all this information won’t be going into a profile. This makes me want to have my own “helper” service.
You must log in or # to comment.
Llama.cpp running qwen 3.8 on a mini PC with 32GB ddr5 ram and 8GB vram. Not the most speedy but it works. I only really access it while at home but I’m also set up with tailscale.
So you’re using CPU offloading? Or the model’s Bonsai2 variation?
I wouldn’t mind running a model at home, mostly to help document my setup. But I just don’t trust it. So no.
It depends on your hardware. But right now the “cheapest” hardware you can buy for a usable setup, is the strix halo (128GB), and run gufo with qwen 3.8 flash next (Q4XL). It is sonnet 5 level. Very good speed (both prefill 1000+ and decode 50+), with up to 256k context.
I have it set up with omp + searxng. Sometimes with hermes. Really good! It changes the game for local setup.
strix halo (128GB)
googles
Jesus christ
Just get the cheapest OEM, Bosgame M5. I know “cheapest” is in quotes. But again, it is the cheapest 128GB unified vram device you can get right now. I got mine at pre-orders, so yes, more than a year and a half ago, prices were a steal. Also, great desktop (mini) PC overall and very efficient, even if you don’t want to do AI.
I have like 150GB of DDR3 spread over multiple 2/4/8/16GB sticks, I wonder if I can get that working for something like this 🤔
Why wouldn’t you trust a local model? I mean sure they are less intelligent but if you dont expose it publicly you should be good to go
I don’t trust it to be accurate; I’d have to reference the sources it pulled from. Extra work for little payoff.
Got it, makes sense. You’d probably have to do some benchmarking first. Depending on how large the source catalog is.
Not only do I run and share it. I’ve built an entire service to allow everyone else to do so safely. https://aihorde.net/
A quick glance shows that I’m in the minority. I grabbed a Bosgames M5 with 128GB UMA RAM when I discovered openclaw (at the time claudebot). I’m running Lemonade, which is a wrapper for llama.cpp, sd-cpp, flux, and others. I’m happy to have paid a “mere” $2400 compared to what similar systems cost now, and now I’m toying with fine tuning my own models. Though, I primarily run Hermes Agent paired with Qwen3.6 35b A3B (using a heretic ablitieration for 0 refusals).
Hermes Agent does a lot of random stuff for me right now. It manages my calendar and to-do list in nextcloud via MCP server. It created and maintains an llm-wiki based on my Obsidian Vault. Then it creates a report every morning about both and the weather.
The server has tailscale installed, so yes I can access it anywhere.
Try gufo with qwen 3.8 flash next. Much much better than 3.6-35B.
How well does it handle concurrency? I could see this as a vast improvement on llamacpp, but I was already speccing out vLLM.
What a timely comment.
I recently gave my kid a gaming PC for his birthday…which I used as an excuse to upgrade my GPU from a 1660 Super to a 5060ti 16GB
Combined with a 5600x3d and 32gb of system memory.
I just started messing with local LLMs yesterday. “Getting started” is rather easy.
pacman -S lm-studio(I use arch, btw), launch it, download a model that your hardware supports, and start talking.Right now I’m working on getting it to talk to vscode.
I used Cline as my VSCode chat plugin for models in LMStudio. Worked pretty well but I have moved my local LLM stack to Hermes in the meantime.
I tried using zoo with qwen3.6 and 2.5 but it failed horrendously. My card could only handle like a 16k context window with those models, and the prompt alone was like 11k.
Tried opencode and that was fast…but can’t get it to interact with system yet. More playing to do.
I just bought a 32GB M6 Mac Mini that I’ll also use for local ai. Will probably use Qwen3.8-27B. Accessing it from outside my lan would happen via Tailscale.
If you are playing around at home I suggest seeking a MoE model. You get less penalty from having less vram and offloading to normal ram. I run Qwen 3.6 quant 6 on 250k context on a 16gb vram 64gb ram ddr4 PC. I get about 31tps which is enough for my use. After a little tweaking I have put results of it’s output into Claude Pro for vetting and it hasn’t found any flaws in the simple scripts and things I’ve asked it to make. I can only run Qwen 3.8 which is a dense model and not MoE. At low quant with not as much context. It spits out accurate short context answers if that’s all you need. So I prefer 3.6. Tiel is faster but Claude picked up more errors in what I was doing. Orinith kept getting into thinking loops for me so I stopped using it. Anyway I’m just a little guy.
I use Qwen 3.8 27b Q4 with 100k context (llama.cpp and the Mistral AI harness) on a 24 GB AMD Radeon 7900 XTX and it’s been great. It can handle pretty much anything I throw at it and I can let it work indepedently overnight / over days on long tasks.
I have nearly the same setup, except OpenCode for the harness. I run 200k context for anything that doesn’t need vision, or 150k otherwise. Typically get about 30 t/s decode With MTP, starts at close to 800 t/s for prefill and drops to the 200s for a long context. I’d imagine higher decode speeds are possible with some tinkering, but I haven’t gotten around to it.
I get about 45 t/s after some tweaking, getting the thermals under control helped a lot; mainly by reducing the power consumption to 261 W (and downclocking a little bit).
What vision model do you use?
How much electricity does it burn? And how much does it cost to let it run overnight? I feel like this is the same problem that the companies have, its way too expensive to use it on a lots of things and it is way cheaper to use it on someone else’s server…
It uses 261 W, probably more expensive than getting a 2nd Mistral subscription with the cost of power where I live (Belgium), but it will be free heat once I have solar panels and a battery.
Thank you, that is not as bad as I expected. 👍
Even ignoring all of the functional issues with LLMs/“AI” related to their design to never admit to not knowing and preferring to make stuff up instead (hallucinate), I have an ethical qualms about the way they are trained. Two primary issues are that they aggressively scrape data from sources that explicitly do not consent to sharing their work and the data centers which have made hardware impossibly expensive for everyone but them and are so impactful to our underfunded electrical grids that they cause brown outs that keep people from heating their homes among other things, cause major spikes that cause house fires and destroy home appliances, and create an excuse to generate power using the worst polluting generators that we had finally gotten shut down to slow climate change (personally I’m tired of the wildfires and extreme storms from climate change).
So, I personally consider it extremely unethical to use the technology for that and several other reasons which are hard to touch on exhaustively in a post like this.
I self host both at home and at work.
At work (for completion and generation) we have a now quite old A6000 running mainly Qwen 3.6 (though I’m trialling 3.8 from Monday) with OpenClaw as the harness. The use case is mainly triage and pre-work on bugs. We have an agent which can be given an exception report with a bunch of information like stack trace and local variable values and it will go off and gather the required code, order files, and data to analyse the issue and come up with a fair idea of the cause and possible solution. This saves devs and support team time. Also looking at adding a PA agent to this which will be able to read emails and calendars and be able to set up (with approval) meetings and move things around and generally make someone’s life a bit less admin-y without having to hire a full PA role.
Also technically in the AI space we have a fleet of Pis with Hailo AI hats running object and anomaly detection in visible and LWIR spectrums for QC
Are you planning on using a specific package for your PA agent? Or is this something you plan on developing in house?
Using openclaw for the harness and then attaching office 365 mcp with an Entra ID App token to give it access to email, calendar, and teams. The MCP format means that we can explicitly control what it reads and writes
24GB GPU
Qwen 3.8 ( 2 variants )
Pangolin>OpenWebUI>lama.cpp for general use and available outside the homelab
Zoo Code>lama.cpp for coding , mostly scripts and html
Qwen 3.8 was a game changer for me.
What variants? And why two?
For coding with large context: Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp Anything else: Qwen3.8-27B-GGUF:Q5_K_M
Not me. Call me old-fashioned, but I prefer organic.
I tried, but didn’t want to invest into it, so I used my 12-GB-vram gpu but no matter what I did it was worse than Lumo from proton,which is basically only one LLM from 3rd party that I am okay to use on occasions and since it has really good free tier I gave up on self hosted LLM.
Yes, I have a desktop with a radeon 9700 ai pro with 32GB of VRAM. Its more than doubled since I bought it but its basically a down clocked 9070xt with double the VRAM. I also have a little arc b50 in my server.
Mostly yeah use it for privacy reasons. I run llama.CPP directly and access via open webui. I host a searx instance for it to use as web search. Mostly run qwen3.6/3.8 or gemma4 on the radeon and almost exclusively gptoss 20b on the B50 along with a second small model for tags and titles in open webui. Yeah I just use it mostly as a basic chatbot or as an agent in obsidian using the engram plugin. Helps search and summarize years of notes and works pretty well.
I’ve tried to use it as a coding assistant in zed without much luck so usually just post my code into open webui for feedback and it works alot better for me.
Yeah I can access it with other machines on my network including phone with conduit app. I can get to it on the go by connecting to home network with wireguard and sending a magic packet to wakeup desktop.
I havent done any image gen nor have much use for it outside of maybe using it to make app icons in the style of papirus icons for programs they don’t and won’t support.
It definitely feels more comprable to first ai wave than current frontier models but the privacy and price trade off is worth it to me. I’d say using it in obsidian has been the best use of it so far and one I certainly wouldn’t trust to an external AI.
Hardware:
2 x ASUS GX10 128GB(each)
2 x Mac Mini M4 Pro 24GB(each)
The GX10s are clustered and run DeepSeek-V4-Flash-0731-Abliterated. The M4 Pros run various utility LLMs such as Qwen 3.8
Software:
OpenWebUI
Hermes Agent
LiteLLM
Llama.cpp
vLLM
StoryBinder - custom web app that I use for story/script writing when in the mood
LASM - Local AI Stack Manger - custom web interface to manage pretty much all aspects of my local stack. Agents run on the 4 nodes as well as my NAS(Download GGUF/HF snapshots once and copy out to the nodes)
Have yet to connect my Home Assistant into the stack.
For mobile access I use Conduit https://github.com/cogwheel0/conduit