ChatGPT Can Guess What You Never Told It. New Study Proves You Can't Stop It.

A new academic study has confirmed something most marketing teams have not priced in: large language models like ChatGPT can accurately infer your age, location, occupation and even relationship status from text that mentions none of those things. Worse, when 240 people were asked to rewrite that text to block the guess, they succeeded only 28% of the time.

The paper, presented in the ACM Digital Library, moves the privacy conversation past the thing everyone worries about (data leaks) to the thing almost nobody has planned for (inference). I think this is the most important AI privacy finding of the year for anyone who touches customer data.

What actually happened

Researchers ran a survey with 240 U.S. participants. Each person read short text snippets drawn from SynthPAI, a synthetic dataset built for personal-attribute inference research, and was asked to do three things.

First, estimate which personal attributes an LLM could infer from the text. Second, report how concerned they were once the true inference was revealed. Third, rewrite the text to block that inference while keeping the original meaning.

Their rewrites were then benchmarked against rewrites produced by ChatGPT itself and by Rescriber, described in the paper as a state-of-the-art sanitisation tool. The results are uncomfortable reading.

The findings, in the study's own figures

People are bad at spotting inference. Participants performed only "a little better than chance" when guessing what an LLM could deduce. In my experience, that is the whole problem in one line: you cannot defend against a risk you cannot see.

Concern arrived late. Nearly half of participants expressed concern once the inference was revealed to them, but concern levels did not vary strongly by attribute type. People were not more worried about location than occupation. They just had not thought about it at all.

How the rewriters compared

Caption: The model that creates the inference risk was also the best at neutralising it, outperforming both humans and a dedicated privacy tool. Source figures: ACM study, 3772318.3791762.

There is a real tension here. The tool people would reach for to protect themselves, Rescriber, was less effective than their own untrained attempts, while the model driving the risk was the strongest defence. From my observation, that dependency is exactly the kind of thing that ends up in a regulator's crosshairs.

Why paraphrasing fails and abstraction works

The study analysed the strategies people used. Paraphrasing was the most common approach and also the least effective. Swapping words around leaves the underlying signal intact, so the model still lands the guess.

What worked better was abstraction and adding ambiguity. Instead of rewording "my morning commute into the city," you generalise it to "my journey to work," removing the geographic and lifestyle signal rather than just rephrasing it.

That distinction matters because it explains why so much AI privacy tooling underdelivers. Most sanitisation targets explicit identifiers, names, emails, phone numbers, the stuff users type in. It does nothing about what the model can quietly deduce.

Why this matters for marketers and growth teams

Context first. The paper notes ChatGPT reached an estimated 800 million weekly active users by April 2025, with a daily average of 187.91 million visits. Inference at that scale is not an edge case, it is the default condition of modern digital behaviour. I have written before about how ChatGPT's audience is no longer who most brands assume it is.

Here is the practical concern. Every time your team pastes customer transcripts, support tickets, survey responses or CRM notes into an LLM, you are not just risking the explicit PII you remembered to strip. You are handing the model everything it needs to infer attributes you never collected and never had consent to hold.

This could create a real compliance gap. You can honour a data-minimisation policy on paper and still feed a model enough context to reconstruct sensitive profiles. That is a governance problem, not a tooling problem.

What to do about it

1. Audit what you paste, not just what you store. Treat free-text customer language as inference-rich, because it is.
2. Abstract before you prompt. Train teams to generalise context ("a customer in the north west") rather than paraphrase it. The study is clear that this is the strategy that actually works.
3. Do not trust sanitisation tools blindly. Rescriber underperformed untrained humans here. Validate any tool against real inference, not just PII flags.
4. Write inference risk into AI governance. Make it a named clause, not an afterthought.

If you are still building the underlying discipline, my view on why your data foundation matters more than your vendor list is a sensible next read.

The takeaway

In my opinion the headline is not that ChatGPT is invasive. It is that people cannot tell what they are giving away, and the tools sold to protect them are not reliably better than guessing. For any team routing customer language through an LLM, inference-aware handling has just moved from nice-to-have to non-negotiable. Start by assuming the model already knows more than you told it, because this study proves it usually does.