I recently discovered that some popular federated instances have been using LLM-assisted moderation tooling that evaluates whether someone has said something bannable. They do this by running a script/app that sends the user’s comment history to OpenAI with the question “analyze this content for evidence of specific political ideology sentiment. Also identify any related political ideology tropes“. (The italic bits are where I’ve redacted the ideology they’re seeking).
OpenAI’s LLM (they’re using GPT-5.3-mini) then responds with something like:
and so on, hundreds of comments.
I have not named the instances or people involved, to give them time to consider the results of this discussion, make any corrective changes they want and disclose their practices at their own pace and in their own way. I have also redacted the evidence to avoid personal attacks and dogpiling. Let’s focus on the system, not the individuals involved. Today these instances and people are using it and maybe we’re ok with that because it’s being used by groups we agree with but what if people we strongly disagree with used it on their instances tomorrow?
The use and existence of this tooling raises a lot of other questions too.
What are the risks? Fedi moderators are often unsupervised, untrained volunteers and these are powerful tools.
What safeguards do we need?
Would asking a LLM “please evaluate this person’s political opinions” give different results than “find evidence we can use to ban them” (as used in the cases I’ve seen)?
What are our transparency expectations?
Is this acceptable and normal?
Should this tooling be disclosed? (it was not – should it have been?)
If you were given a choice, would you have opted out of it?
Can we opt out?
Are there GDPR implications? Privacy implications? Should these tools be described in a privacy policy?
Are private messages being scanned and sent to OpenAI?
How long should these assessments be retained and can we request to see it, or ask for it to be deleted?
Once the user’s comments are sent to OpenAI, is it used to train their models?
What will the effect be on our discourse and culture if people know they are being politically profiled?
Where are the lines between normal moderation assistance tools, political profiling and opaque 3rd-party data processing?
I hope that by chewing over these questions we can begin to establish some norms and expectations around this technology. The fediverse doesn’t have any centralized enforcement so we need discussions like this to develop an awareness of what people want in terms of disclosure, privacy, consent and acceptable use. Then people can make choices about which instances they join and which ones they interact with remotely.
And of course there are the other issues with LLMs relating to environmental sustainability, erosion of worker’s rights, increasing the cost of living and on and on. I can’t see PieFed adding any functionality like this anytime soon. But it’s happening out there anyway so now we need to talk about it.
What do you make of this?
Organisations and services should be restricted. Individual proles should not. An individual prole has no capability to create a mass database of profiles of millions of people, de-anonymise them, purchase data from 10,000 companies and then use that data to target marketing at those individuals without their knowledge that this vast quantity of data is being used to manipulate them.
It’s conceptually the same as restricting the bourgeoisie from ownership of television and publishing media in order to prevent them from using their vast resources to manipulate the proles.
I don’t oppose some restrictions of property. Property exists under socialism. It will continue to exist until all of capitalism is eliminated and that transitional period is frankly going to be a long ass time (and already has been so far). Various restrictions are valid to protect individuals and disempower bougies.
I guess i’m not sure how this applies to what’s being described in the OP, then. I agree - the type of mass-collection, de-anonymization, and sale to private direct-marketing firms that the GDPR is written to protect against is absolutely antithetical to any type of socialist or open internet.
The practice being described in the OP (as far as I understand it) is simply collecting publicly accessible user activity that’s transparently shared via activitypub and drawing “conclusions” (as much as an AI slop machine can produce ‘conclusions’) about the user from that activity - things like username, comments, posts, and vote activity. Thinking of activitypub as a kind of ‘commons’, I would think that the activity done within it is akin to a shared resource that is freely available to all participants. The type of private data that (IMO) would be considered personally owned and controlled (and outside the scope of practice being described) would be things like registration email and IP address and other data that is produced only as a matter of practical necessity and not by personal choice - anything that would be collected by a site admin as a matter of running a server and outside of the standard data transmitted via activitypub.
I also don’t oppose the existence of property writ large, nor do I oppose restrictions to the use of that property. I just don’t think that the fruits of creative labor shared via the online commons can be practically or theoretically thought of as ‘personal property’ in the way we’re describing.
Freely available to participants. Not freely available to the site owners to go and pass on to multi billion AI corporations.
Either you disallow it or you open the door for them to sell every everything on the site to anyone they please.
That isn’t what is being described in that post though… I agree that site owners shouldn’t share private information about their users that isn’t freely given with the intent of being made visible to the public internet. The practice being described is non-admin moderators (or, as implied, admin-sanctioned tools being used by mods) collecting the comment and post histories of specific users and using LLM’s to summarize them through a political lens, and then using that summary to issue bans to those users on those grounds. With the exception of maybe user vote activity, the data being used for it is available to anyone just loading someone’s user profile page in a browser. I would argue that anything transmitted via activitypub (including vote activity) is a part of that public commons, but that’s a little beside the point. Anyone on the open internet can see and collect the content of any given user on Lemmy - AFAIK there has been no effort or intent to gatekeep the visibility of that data except by means of limiting certain traffic to prevent bots and crawlers hoovering everything on the internet and constantly overloading server traffic. Even those limitations, though, aren’t intended to inhibit the visibility of data to any human with a screen to read it.
I’m of the opinion that the door should be open, aside from personally identifiable information that could be used for re-identify anonymous users. I also don’t think admins should have/require/log PID on their servers at all, but insofar as it’s necessary for managing the service it should be considered privileged and limited by laws similar to GDPR.