One of the most disturbing tech news opening 2026 is the flood of non-consensual intimate imagery generated through Grok. The ensuing massive backlash this generated in the public everywhere is only natural. In Europe, this has prompted Members of Parliament to take a bipartisan stance to ask the Commission to ban this practice through the AI Act, citing the DSA as not sufficient to tackle the issue. The linked article by Politico reports that while X committed to stop users from “nudifying” pictures, it was apparently still possible to do it in the UK and Europe alike. The upcoming negotiations for the Omnibus package involving the AI Act offers a good opportunity to have a serious debate around this issue. Indeed, the newly appointed Rapporteur for the file on behalf of the LIBE Committee, Irish Renew Member Michael McNamara, announced his intention to explore including a ban of this sort in the package. In this he aligns with the position taken by his own country’s AI Advisory Council.

I would like to offer some additional context to this proposal and see whether and how the AI Act can tackle the issue now and in perspective.

First of all, as we all know, this is not the first time this happens. Maybe the scale of it is particularly striking, but the issue of non-consensual intimate imagery facilitated by AI deepfakes has been there for a while, so much so that even Congress managed to pass a law on it, which is basically all they could do in regulating AI at federal level thus far. The Take it Down Act, strongly supported by Melania Trump, includes both platform-related removal orders (that take effect in May) and complex updates to criminal law for whoever creates such pictures.

The AI Act does not work like this. This leads me to the first time we had a serious debate about this. It was during the trilogue negotiations, the 6-month long final stretch to the final deal. There had been a striking case in Spain just around that time, and Spain, which retained the rotating Presidency back then, was interested in acting on it. At that time, though, this didn’t take the form of a possible ban.

Bans were considered almost toxic, especially by the Member States, and it would have taken a massive cross-country scandal to even have them consider the possibility of going that far. Everybody was treading very carefully around bans, and focusing on the most pressing at the time.

As you know, deepfakes are only regulated through a transparency obligation in the AI Act: they just need to be labeled as such. Not even vaguely considered as high-risk practices. Again, at the time, banning them would have simply been unthinkable.

So what Spain was proposing at the time was not to ban them. Rather, they were proposing to expand fines also to individual users. The Commission even put forward a proposal, having a maximum amount of 15,000€ for individual violations (so, much much lower than what a provider or deployer might face) in such cases.

The proposal saw a massive, bipartisan backlash from the Parliament, already at advisers’ level. Until that moment, only legal entities being economic operators in the AI value chain were ever considered being subject to fines, because they were the only actors involved in the AI Act. There was no individual user, for non-professional reasons. Actually, non-professional uses even by economic operators are explicitly excluded by the Act (well, Spain tried to change this precisely to introduce their proposal), as it is still product safety legislation, which typically does not punish end users. There might be other tools for that, but not this type of legislation.

At the time, I had to motivate the Parliament’s objection and I remember our discussions: introducing fines for individual users of deepfakes or AI generated content for violating article 50 (transparency) would have meant exposing students to 15,000€ fines. I even thought of elder, less tech-savvy people: I would think back then of my own mother caught playing with deepfakes (maybe in this case political ones, certainly not sexual) just because she might misunderstand some implications and being horrified at the prospect of having her slammed with such a steep fine for an individual.

Among our objections were also two more arguments:

1. what would change by just labelling the nudified image as AI-generated? Would that prevent the done harm to the affected girl in any way? Spain responded that at least the girl’s honor would be safe. Not convincing enough to completely overhaul the product safety nature of this law.

Argument number 2: this is a serious issue that should be dealt with by criminal law, which is much more a national competence than an EU one, and as such cannot be dealt with in an EU Regulation. And as such, runs the risk of great fragmentation among Member States, but that’s a different story.

In any case, this attempt by the Spanish Presidency was abandoned, as the time was limited, the opposition was clear and the hot topics were so much more pressing than the thought of introducing new ones at the time.

Now that the MEPs seem to be in favor of adding a prohibition to article 5, let’s see how that could work, because I don’t think it is as straightforward as it may seem.

If you look at the language of all the newly introduced bans (d, e, f, g) [edited after a conversation with Barry Scannel], they all start by “the placing on the market, the putting into service for this specific purpose, or the use of”. I underline the words “for this specific purpose”. How can a GPAI model such as Grok ever fall into this category? Paradoxically, you should thank Spain for this wording. They came up with it during the infamous 36 hour-long final trilogue, on which one day I should write a book, as there are so many things to say about it. They introduced it as a partial counteroffer to the Parliament’s long list of grievances on prohibitions and law enforcement measures. This would make the prohibitions really targeted but also virtually almost unenforceable, because it is simply so easy to circumvent for a provider. The only part that would hold, even in this case, would be on a deployer, except that in this specific case we are not talking about a company or law enforcement (again, legal private and public entities that are subject to the AI Act) but about individuals. Introducing a ban of this sort would reintroduce the need to expand fines to individuals, just as Spain proposed in the first place.

But that’s not all there is about this in the AI Act. Again, Grok is a GPAI model, and one with systemic risk at that (presumably going well above the 10^25 FLOPs threshold). As such, it is subject to the highest tier of obligations for models under article 55 (risk assessment and mitigation, model evaluation, incident reporting, cybersecurity). We also know that xAI, signed the GPAI Code of Practice’s safety and security chapter.

Now, as the Code of Practice was heading towards its final stages last Spring, one of the most contentious issues was the placement of fundamental rights in an “optional list” providers should control for, in the third draft. Non-consensual intimate imagery was among these optional risks:

“Risks to fundamental rights, e.g., risk to freedom of expression; risk to non-discrimination; risk to privacy and the protection of personal data; risk from child sexual abuse material (CSAM) and non-consensual intimate images (NCII)”.

At the time, the drafters argued these risks are not inherent to the high-impact capabilities of the models, and as such are not treated as mandatory to control for.

The backlash from both Parliament and civil society was massive. Even I stepped in and published an op-ed together with two participants to the drafting working groups, Laura Lazaro Cabrera from CDT and David Evan Harris from Berkeley.

The pressure worked in the end and fundamental rights were reinstated in the final text (page 34):

“For the purpose of identifying systemic risks pursuant to Measure 2.1, point (1), and Article 3(65) AI Act, the following distinct but in some cases overlapping types of risks apply:

(1) Risks to public health.

(2) Risks to safety.

(3) Risks to public security.

(4) Risks to fundamental rights.

(5) Risks to society as a whole.

Based on these types of risks, a list of specified systemic risks is provided in Appendix 1.4. As part of the systemic risk identification process that Signatories will conduct, examples of risks falling under the five types of risks above that they will draw upon when compiling the list of risks in Measure 2.1, point (1)(a), are: risks of major accidents; risks to critical sectors or infrastructure, public mental health, freedom of expression and information, non-discrimination, privacy and the protection of personal data, the environment, non-human welfare, economic security, and democratic processes; and risks from concentration of power and illegal, violent, hateful, radicalising, or false content, including risks from child sexual abuse material (CSAM) and non-consensual intimate images (NCII)”.

Now, it is clear that we are in the realm of what is happening now. xAI signed these commitments to assess and mitigate these types of risk. They are subject to enforcement by the AI Office, that so far has been treating signatories as acting in good faith to implement the Code. This episode should be a clear hook to launch an investigation, at the very least, and request measures to stop the model from creating such imagery. This way the AI Act already covers this problem.

Adding a ban would instead tackle the end user-side of the issue, but running into all the problems I described above. I hope legislators tread very carefully in the months ahead.