Controversial AI image platform Civitai dropped by its cloud computing provider after reports of possible CSAM

OctoML has ended its business relationship with Civitai following an investigation by 404 Media, which revealed that Civitai’s text-to-image platform was being used to generate images that potentially constitute child pornography. The investigation showed that Civitai users were creating sexually explicit and nonconsensual images of real people, including pornographic depictions of children.

Civitai and OctoML had introduced new measures to prevent the creation of harmful images, such as filters to block the generation of NSFW content and mandatory embedding to bar the model from generating images of children if mature themes or keywords were detected. Despite this, OctoML decided to cut ties with Civitai, citing its commitment to ensuring the safe and responsible use of AI.

The controversy involving Civitai’s platform highlights the ethical and legal implications of AI technology, particularly in image generation, and underscores the importance of implementing robust measures to prevent the creation of harmful content. This case also raises awareness about the need for responsible AI usage and the potential risks associated with AI platforms that can be exploited for creating nonconsensual and explicit content.