I’ve recently added anubis to lemmy.ml, and it seems to be working well.
I have a PR to add anubis to lemmy-ansible (our main installation method), and I could use some help tweaking / optimizing its botPolicy.yaml config, for federated services.
Anyone with experience running anubis, this would be much appreciated.



It would be nice to have a much more aggressive anti-bot stance for communities/content that aren’t local. If google or any other crawler wants to crawl c/lemmy@lemmy.ml then it should do it on the source instance. Doing it on mine makes no sense.
Well behaved bots should see the origin in the headers and only crawl those.