A court in the Netherlands has ruled that xAI’s Grok AI chatbot cannot create images of people who are depicted in any kind of undressed or sexually suggestive manner without the individuals in those pictures’ express consent. The court’s decision is one of the first major legal actions taken in Europe with respect to misuse of AI image technology.
Landmark Ruling on AI Image Misuse
The Amsterdam District Court has required xAI and Grok to stop either creating or distributing images depicting a person completely or partially undressed without their express consent, adding that content of this nature is illegal and constitutes a serious risk to the population, especially when images of real people are manipulated.
In addition, the judges made it clear that the company must comply with the order immediately, or they will face serious consequences. xAI may be fined up to €100,000 per day for failing to honour the order; in addition, Grok will not be available on the social media site X within the Netherlands or possibly anywhere else if violations of the prohibitive orders continue.
Heavy Penalties and Platform Risks
The action against xAI was initiated by organisations in the Netherlands Offlimits and the Victim Support Fund, to show that Grok enabled significant amounts of abuse online against victims. They claimed to have seen a massive increase in victims and said immediate action must be taken to prevent additional abuse.
According to test results from Grok’s testing earlier this month, it may have been generating explicit or non-consensual images despite xAI’s assertion that it had implemented effective safeguards. During court-ordered testing, it was still producing harmful content even with those safeguards allegedly in place.
Safeguards Fail Under Testing
The ruling is also concerned with the safety of children. The court specifically ordered that Grok should not create or distribute any materials that would meet the definition of child sexual abuse material under Dutch law. The decision reflects widespread concerns throughout Europe about the misuse of AI tools used to create deepfake imagery of both adult and minor individuals.
This ruling is being viewed as a significant moment in the international discussion regarding AI regulation. In recent months, the European Union and other governments and regulatory agencies have increased pressure on technology companies in response to deepfake sexual imagery and the misuse of generative AI. Grok has already been subject to investigation and legal action in numerous countries around the world.
Khalid Qasid is a media enthusiast with a strong interest in documentary filmmaking. He holds a Master’s degree in Convergent Journalism from AJK MCRC. He has also written extensively on esports at Sportsdunia. Currently, he covers world and general news at NewsX Digital.
