A UK regulator questioned a chatbot's pre-launch risk check

What happened

On 6 October 2023, the UK Information Commissioner's Office (ICO) announced a preliminary enforcement notice against Snap over its generative-AI chatbot, My AI. The notice states the ICO's investigation had provisionally found that Snap failed to adequately assess the data-protection risks the feature posed to several million UK users, including 13 to 17 year olds. A preliminary notice is not a final finding; it gives the company a chance to respond before any enforcement decision is made.

What the documents show

The ICO's own statement quotes Information Commissioner John Edwards describing 'a worrying failure by Snap to adequately identify and assess the privacy risks to children and other users before launching My AI.' It records that My AI, built on OpenAI's GPT technology, launched to Snapchat+ subscribers in the UK in February 2023 and rolled out to the wider user base that April, by which point Snapchat had 21 million UK monthly active users. The notice states that if a final enforcement notice followed, Snap could be required to stop offering My AI to UK users until it carried out an adequate risk assessment. A separate ICO explainer published six months earlier, in April 2023, sets out the underlying obligation: that any organisation processing personal data through generative AI must complete a data protection impact assessment before deployment.

The mechanism

The legal mechanism is the UK GDPR's data protection impact assessment, or DPIA, a requirement to analyse and minimise privacy risk before processing personal data in ways likely to pose a high risk to people's rights, which the ICO's April guidance says applies squarely to generative AI launches. The October notice is the enforcement side of that same requirement: a provisional finding that Snap's DPIA for My AI did not adequately weigh the risk to minors specifically, as distinct from a finding that My AI's outputs themselves broke any rule. Preliminary and final enforcement notices are formally separate steps, with representations from the company considered in between.

What it leaves open

The October 2023 notice is provisional by its own terms, and the record examined here does not include a subsequent final enforcement notice, so this is not evidence that a breach was ultimately confirmed or that My AI was ever required to stop processing UK users' data. It also does not assess My AI's actual conversational safety, only the adequacy of the risk-assessment paperwork behind its launch.

• Did a chatbot's developer publish or reference a completed risk assessment before launch, particularly regarding minors?
• Was a regulatory notice described as preliminary, or does the record show a final enforcement decision?
• Does an assessment address the product's risk to a general audience, or specifically to the age groups most likely to use it?

Read together, the two ICO documents connect a general rule, that generative AI needs a proper privacy risk assessment before launch, to a specific case in which a regulator found reason to doubt one had been done properly for younger users.