The “It’s not clear who’s doing it” section is what’s baffling to me, like I’m very far from into networking and I’ve definitely done some crude scraping the few times I’ve needed it, but even I am pretty sure I could do better than this. Makes me think someone with more money than sense hooked up an “Agentic” AI and told it to gather data to train the next AI then just let it decide how – except not because I’m pretty sure the AIs would have more sense. It’s really weird.
Ai scrapers are making it kinda suck to run anything on the internet to be honest.
Making it kinda suck to use the Internet at all. I try to look things up and its page after page of AI slop. We are going to end up back in the days of Altavista.
For some reason, for the past month or so, when I google stuff the first result is ‘Can’t generate an AI overview right now, try again later’. Maybe one time in 50 it’ll do the AI response.
I don’t know if I changed a setting and forgot, or if they just don’t like me, or if their local processing can’t hack it. Either way, it makes me actually go to the websites, like the good old days of a couple of years ago! And I find better info doing that.
Honestly, the AI overview is crap. One time, searching for legal stuff about a potentially precedent setting case I’m involved in, it responded referring specifically to my case - as though it had been resolved and the precedent was set.
So yeah, I wouldn’t trust that shit to provide me a recipe for biscuits.
i am actively looking for a google alternative, they are horrible with their summaries and because it sounds confident, it throws you off and derails actually finding an answer. their search has been seo optimized clickbait for years but now with reddit becoming ai slop and another billionaire persona cleansing space its looking bleak
If you don’t mind paying for it, I like Kagi. There are some people who don’t for reasons outside of its performance, so make sure you do your research if you care about those things.
For free, DDG isn’t bad, but there’s also options that can be self-hosted or you can find other people’s hosted instances: SearXNG and LibreY for example. (I’d link to some but I don’t want to accidentally get them hugged to death since they’re being run by small independent servers.) I don’t use any of the above free options so I can’t comment on which one’s better or not but at least I am aware they exist. I know SearXNG lets you also pick and choose which search engine services are used in your searches.
Ai
scrapersaremakingitkinda sucktorunanythingontheinternetto be honest.
Detect whether the user is a human, but instead of blocking the request (which is going to be obvious to the scraper operator and will just cause the bot developers to go work on better human emulation until they get the data), poison the response. Just as blocking scrapers is hard for website operators, so is separating useful data from not-useful data for people building AI training corpuses.
https://www.cloudflare.com/learning/ai/data-poisoning/
Data poisoning involves injecting malicious information into training datasets to manipulate an AI model’s behavior, compromising its accuracy, reliability, and the overall integrity of machine learning results.
This is the way. Computer use agents are common and can easily ‘browse’ to a page and grab the content.
The really exasperating thing is that Wikipedia makes all of their data available in compressed database form, and I suspect that other MediaWiki wikis could do so too. I checked one of the wikis being complained about, the Minecraft wiki, and it’s a MediaWiki wiki.
This isn’t even a case where the server operators have any problem with the data being available to the bots. In fact, people are going out of their way to package up all the data in a form that is optimal for computer processing (better for the bots) and which doesn’t create additional load on the wiki servers (better for the wiki operators). They’re providing data in delta format!
But the bots aren’t using them! And on top of that, they aren’t even written to follow traditional conventions for being polite in scraping. We’ve had lots of software spider the Web — that’s not new — but normally that spidering software has followed basic politeness to try to avoid excessive disruption to the servers being spidered.
It’s as if someone has set up a table out front of their house with a big sign reading “Free cookies!” with boxes of pre-packaged cookies. Instead of taking the damn cookies, the bots are obtaining the largest trucks possible and then ramming holes in the house with a truck and scavenging through the ruins, desperately trying to find cookies.
It’s a weird phenomenon I encountered as a data engineer inside corporations. Teams wanted to run endless queries off our API instead of using the snapshots we provided. Most of the apps didn’t have a justifiable reason for wanting up to the minute information. 🤷♂️
That compressed database would have to be updated frequently, I don’t know how well it’d work.
Wikipedia has started striking deals with AI companies as a means to cover the cost of their lookups which, as much as I may not like it, is totally fair.
A wiki for some obscure indie game won’t have the same leverage.
That compressed database would have to be updated frequently
They are.
Anubis is the way to go
Wait a second…I already read this… 2 months ago