The problem with AI data studies.

Every week there’s a new AI search study tracking 100,000 prompts, 500,000 citations, and millions of responses.

Out of those come the conclusions:

But there’s something wrong with this.

The problem isn't necessarily the data. It's what marketers assume the data proves.

Most of these studies are accurate to the test that was run. But that test is impacted by which prompts were chosen, which model was used, where it was run, how the prompts were worded, and how many times each one was tested.

Change those inputs, and you can get a different result.

So when a study says Reddit wins 16% of ChatGPT citations, that doesn’t mean Reddit accounts for 16% of citations across ChatGPT.

It means Reddit accounted for 16% of citations within that particular dataset.

That is the distinction I think marketers are missing.

A bigger sample can make the result more reliable for that test. It doesn't make the conclusion universally true. It also doesn’t mean the conclusions apply to your AI search strategy.

I started looking into this after the recent studies about Reddit and how ChatGPT suddenly stopped citing it.

Ahrefs currently has Reddit at 16.8% of ChatGPT citation share across its broad US dataset.

Promptwatch measured Reddit at 3.83% of ChatGPT Search citations before seeing that fall to just 0.52% during part of August.

Those numbers are not reliable enough, and both draw different conclusions.

Why the gap?

So saying: “ChatGPT gets 16.8% of its citations from Reddit and is the most cited platform” is already misleading if marketers are taking that as a hard truth.

A more accurate interpretation is: Reddit accounted for 16.8% of citation share inside this particular dataset.

It’s less exciting, especially when we don’t have a great deal of AI search data ourselves, but it's not misleading.

Because if you're deciding whether your company should invest in Reddit, going by these data studies can send you down the wrong path.

There’s no public Search Console for ChatGPT.

Google knows what people search because they have the query logs.

AI tracking companies generally don't have a complete record of every question people ask ChatGPT, Gemini, or Claude.

So before they track anything, they have to decide: Which prompts should we test?

Those choices determine everything that comes after.

For example, Ahrefs.

Its Brand Radar corpus starts with its Google keyword database and Google's People Also Ask questions. It then expands those into related questions and runs them through AI platforms.

Ahrefs is open about this and describes its metrics as ‘modeled signals’ rather than audience measurements.

Its ‘AI adjusted volume’ makes that clearer. Ahrefs doesn't know that a particular ChatGPT prompt was asked 150 times.

It takes the Google search volume of a related parent keyword and applies an estimated ChatGPT ratio.

So something presented as estimated AI demand ultimately starts with:

Google search demand.

It’s still helpful as a baseline understanding, but it doesn't show how many people ask ChatGPT that question.

If a tool tells you a prompt has high AI demand, I wouldn't start prioritizing content because of that number.

First you have to think: Is this observed AI behavior or modeled demand?

Because you're making two different decisions.

One says: People are asking this inside AI.

The other says: Based on another dataset, we think this is probably an important AI question.

Both can be helpful, but they’re not the same evidence.

This is probably the biggest issue with AI data studies.

Sample size is the main focus - “we tracked 10 million prompts…”

But sample composition is more important here.

For example, if I were to look at project management software.

I can test:

Or:

They're all valid questions about the same topic. But they're likely to produce completely different brands and sources.

There's evidence for this too - Semrush tested 115 prompts across 14 countries and found that short conversational prompts produced 30-50x more brand mentions than longer prompts.

Comparison queries generated 2.4x more brand mentions than informational ones.

The market didn't change but the questions did.

So when a study says: “We analyzed 500,000 citations and found that X is the most commonly cited source.”

My first thought isn't: How many citations?

It's: What did you ask?

Because 500,000 responses to generic questions will tell you a small part of what happens when your actual buyers start adding:

A large dataset can be reliable at describing its own sample, but that doesn’t make the sample representative of your customers.

Now some tools are more accurate to actual AI behavior.

Profound, for example, uses anonymized conversations from opt-in AI users rather than turning search keywords into likely prompts. Which is stronger evidence of how people actually use AI.

But it’s still a sample.

Profound applies statistical modeling to correct bias, and its industry Index clusters conversations into representative prompts while removing some brand-specific and regional queries.

That makes sense for comparing companies within an industry. But it can also remove questions such as: HubSpot vs Salesforce for a 30-person SaaS team

Those are often the questions that are more important to an individual company.

So even billions of real prompts don't tell you what your buyers are asking or what they see.

I wouldn't ignore these studies completely. I'd just stop taking the headline findings as rules that apply to all scenarios.

The studies are still helpful. You just need to be clear about what they measured before changing your marketing because of them.