Maybe. 2nd idea I’ve got is that if no one is replying after say 24hrs and something like 75-80% of your posts are as such and you have at least 100 such posts, you get added to the list?
Main concern I see about something like this is false positives and how someone real could end up getting blocked.
I definitely want to think on this some more but it might have some legs.
…I wonder if there’s a programmatic way to detect these bots? Some sort of analysis on their posting behavior?
If they’re playing nice they’ll have the bot flag checked in their profile, and then maybe build a list of any bot that creates posts? As most of the “good” bots just reply to comments? Anyway just thinking out loud. But I’m thinking I could easily add a public API to my search engine that just returns a list of “posting bots”…
The target folder may be quite large. You can look at the dependencies for my project but my end binary is only a few MB.
I don’t see anything wrong initially. Can you also test with something like postman? It’ll allow you to control the headers etc… that you send.
You can also compare to what I have here and see if that helps: https://github.com/marsara9/lemmy-search/blob/b6c88355aba49abca52862473650526821ee165a/server/src/api/lemmy/fetcher.rs#L123
Thanks for the callout. For anyone using that site, I have a list of bugs I’m hoping to get fixed in the next couple days that should fix a lot of the major issues (broken links, etc…).
HTML + JavaScript frontend. Rust backend with a postgres database.
It’ll be open sourced once I can get the MVP ready.
Would the canonical tag make any sense for Lemmy? The problem is, if you search for something your preferred site / URL is your instance. So the canonical would be different for every user?
I’m using the public API to grab every post / comment and then I essentially replace the content with only the unique words. Then when you go to search it just looks for any post or comment, in my database, that has the words you typed in. Finally I sort based on the number of upvotes.
Right now it only craws a specific instance that you point it to. But as long as that instance is federated it /should/ get everything. But eventually I plan on using that instance’s list of federated instances to scan everything and lighten the load on any one particular instance.
Edit: I thought about tapping into the existing database but the existing database is more geared towards serving content but not necessarily searching. The database that I’m building you can search but I drop so much of the original data that using it for content is worthless.
I’m doing tests in the next couple days. But I’m trying to build a search engine specifically for Lemmy.
I’m hoping I can open it to the public in a week or so.
Problem is finding the difference between repost bots and bots that are helpful like automod and link redirectors.