Lovely #fediverse does anybody have a list of all the known user-agents for #activitypub software ?
-
@smallcircles that's what I'm thinking. I see a lot of random go and other default package UAs, some are legit fediverse servers and others are skriptkiddies that zone will always be very gray.
I guess the best is a little bit of a baseline to work with and the rest will come out in the logs.
I'll check out your project, it looks interesting
@kc @smallcircles There are quite a few instances out there that will refuse to answer requests unless they have a non-default User-Agent. I think it's generally done by blocking known defaults, rather than allowing known non-defaults.
-
? Guest crossposted this topic to General Discussion
-
@kc @smallcircles There are quite a few instances out there that will refuse to answer requests unless they have a non-default User-Agent. I think it's generally done by blocking known defaults, rather than allowing known non-defaults.
@FenTiger @smallcircles normally yes, my WAF is paranoid psychotic particularly on aws, google cloud, azure, and digitalocean (99% of the internet's problems come from these ranges).
Problem here is that there are instances that exist in those IP spaces, that's why I'm tweaking in user agents and a couple of other checks before it hits the line "fuck-bezos-gates-and-zuckerberg"
-
Lovely #fediverse does anybody have a list of all the known user-agents for #activitypub software ?
I have some WAF rules to write to make sure my server can be seen behind a frontend that is only meant to battle AI crawlers and I rather not (knowingly) leave endpoints open to OpenAI
@kc nope, but the UA of AI bots are known, so what I am doing is, to block them, instead of allowing "everything else".
The other reason I suggest you to block them, is that I am developing my own instance Aktor , and then you will have a new UA. And the fediverse is expanding, with more and more coming.
Better if you focus on blocking AI bot UA. -
@kc nope, but the UA of AI bots are known, so what I am doing is, to block them, instead of allowing "everything else".
The other reason I suggest you to block them, is that I am developing my own instance Aktor , and then you will have a new UA. And the fediverse is expanding, with more and more coming.
Better if you focus on blocking AI bot UA.@uriel I wish it was that easy, you'll be surprised how many "legitimate" users, UAs, and retail IP ranges are currently eating shit in my tarpit.
The obvious ones are blocked on three levels at this point, I'm all about making new problems for the smarter ones (and for myself, obviously)
-
@uriel I wish it was that easy, you'll be surprised how many "legitimate" users, UAs, and retail IP ranges are currently eating shit in my tarpit.
The obvious ones are blocked on three levels at this point, I'm all about making new problems for the smarter ones (and for myself, obviously)
@uriel the idea is more understand what we know of in the fedi, open the doors to them, grab the new ones every so often from the logs to adapt, while not letting the capitalist pigs eat all the food in the room because the door wasn't closed enough
-
@uriel the idea is more understand what we know of in the fedi, open the doors to them, grab the new ones every so often from the logs to adapt, while not letting the capitalist pigs eat all the food in the room because the door wasn't closed enough
@kc Sorry, I am not into crusades, of any sort.
-
@uriel the idea is more understand what we know of in the fedi, open the doors to them, grab the new ones every so often from the logs to adapt, while not letting the capitalist pigs eat all the food in the room because the door wasn't closed enough
@kc@social.coop my understanding is there is no known standard for UA.
https://github.com/NodeBB/NodeBB/blob/87583bb5c70e11274655face9cb7f2a274f2e1f8/src/request.js#L25
NodeBB just sends
NodeBB/ ()Pretty personalized for NodeBB
-
@julian accidental standard perhaps, it looks like most start (PACKAGE)/(version 0.0.0) - eg Plume/0.7.2, Mastodon/(a little bit more), Pixelfed/…
That is a good starting point for me to test and see what sticks. Thanks !
-
@julian accidental standard perhaps, it looks like most start (PACKAGE)/(version 0.0.0) - eg Plume/0.7.2, Mastodon/(a little bit more), Pixelfed/…
That is a good starting point for me to test and see what sticks. Thanks !
@kc@social.coop oh that's just the recommendation described here
https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent
-
@julian seems a lot of people followed it when developping
I caught a couple of randoms (I totally forgot about Friendica) in the bot validator after deploying the new rules, looks like it's safe enough to push in that format to avoid filtering out the fediverse
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login