AI Governance

Humans may read your AI chats. What that means for your data

Praeferre ResearchPraeferre Insights Team
Published
Read time5 min read
AI Governance

Two stories in the week of 14 September 2026 should prompt every organisation to revisit its generative AI policy. 404 Media reported, as covered by The Next Web, that OpenAI uses hundreds of contractors to read and rate real ChatGPT conversations. The same day, Apple released iOS 27 with a new prompt asking users to let Siri interactions help train its AI models. Neither story is necessarily unlawful. Both show that prompts typed at work can travel much further than employees assume.

What was reported

ChatGPT conversations reviewed by people

According to The Next Web’s summary of 404 Media’s reporting, an OpenAI programme codenamed Project Lily uses hundreds of contractors to read real ChatGPT conversations and rate the chatbot’s replies. Reviewers do not see usernames, but can see a summary of the user’s saved memories. OpenAI says a privacy filter model removes personal information before review, while acknowledging the filter can make mistakes, miss uncommon identifiers and under-redact.

The report says the “Improve the model for everyone” setting is on by default for Free, Plus and Pro accounts and off by default for Enterprise, Business and Edu accounts, and that switching it off applies to new conversations rather than those already eligible for model improvement.

Apple’s new opt-in for Siri training

AppleInsider and heise report that iOS 27, released on 14 September, changes Apple’s privacy wording. Where it previously said personal data and interactions were never used to train its foundation models, it now adds “unless you explicitly choose to help improve them”. Users are prompted to opt in; shared data can include Siri audio recordings and transcripts, and heise reports that review personnel may examine some recordings. The setting can be changed later under Analysis & Improvements.

Key points
Consumer vs business accounts
Reported defaults differ: model improvement is on by default for consumer ChatGPT plans and off for Enterprise, Business and Edu.
Opting out
Reported to apply to new conversations, not those already eligible for improvement.
Automated redaction
Helpful, but acknowledged by the vendor to be imperfect.

Why this is a business risk, not just a consumer privacy story

Most organisations now have staff using generative AI, whether sanctioned or not. When someone pastes a customer complaint, a contract draft, source code or an HR case into a personal account on a consumer AI tool, three things can happen that the organisation never approved:

  • The data may be used to improve the model, depending on account type and settings.
  • A person outside your organisation may read it as part of quality review.
  • You may be unable to retrieve or delete it in a way that satisfies a data subject request or a client confidentiality obligation.

Under the UK GDPR and EU GDPR, disclosing personal data to a third party without a lawful basis, transparency and appropriate contracts is a compliance failure, regardless of whether the employee meant well. For regulated sectors, confidential client information raises further obligations.

“The safest prompt is the one that never contained the sensitive data in the first place.”

— Praeferre analysis

What to do now

  1. Provide approved business-grade AI tools. Staff turn to personal accounts when there is no sanctioned alternative. Business tiers typically offer stronger data controls and contractual commitments.
  2. Check vendor settings and terms. Confirm training, retention and human review terms for every AI service you allow, and document them.
  3. Publish a clear, short AI use policy. Say which tools are approved, which data must never be entered, and why.
  4. Control data at the point of entry. AI Data Leak Protection detects personal and confidential data in prompts and redacts, obfuscates or blocks it before it reaches ChatGPT, Claude, Gemini or other tools.
  5. Manage corporate devices. Review how new on-device AI features and data sharing prompts are configured on managed phones and laptops.
  6. Record it in your governance framework. Include AI tools in your records of processing, DPIAs and supplier assessments.

Praeferre’s responsible AI advisory helps organisations set practical AI policies and governance, and our AI Data Leak Protection puts those policies into effect at the moment it matters.

Stop personal and confidential data reaching public AI tools, without blocking productivity.

See AI DLP

Common questions

According to reporting by 404 Media, summarised by The Next Web, OpenAI uses contractors to review some real conversations to rate responses, after a privacy filter attempts to remove personal information.

The report says the model improvement setting is off by default for ChatGPT Enterprise, Business and Edu accounts and on by default for Free, Plus and Pro. Always check the current terms for the plan you use.

Combine approved business-grade tools, a clear AI use policy, training, and technical controls such as AI data leak protection that detect and redact or block sensitive data in prompts.

Sources

  1. Hundreds of contractors are reportedly reading real ChatGPT conversations — The Next Web, 14 September 2026
  2. Apple has altered course on using customer data to train its AI — AppleInsider, 14 September 2026
  3. Backflip: Apple now wants to train AI models with user data after all — heise online, 13 September 2026