How to use Cloudflare for AI search insights.

AI search is harder to measure than organic search.

In search, you can look at rankings, clicks, and landing pages.

In AI, marketers are left with expensive tools, messy data, or just a guess.

But Cloudflare AI Crawl Control is holding data that could be valuable to understanding AI search for your brand.

It uses server log data, which, if you don’t use Cloudflare, can be accessed wherever your hosting provider is.

A quick caveat: Cloudflare only offers 24 hours of data (export daily), but the paid version is still cheaper than AI tools. Or just use your server logs directly for free.

Here’s the full article on this.

Log into Cloudflare > select your domain > open AI Crawl Control > go to the Metrics.

Scroll to the bottom and find Paths, Patterns, and Status distribution. Everything else is context.

This dataset is one of the few direct signals you can get from AI systems without paying for aggregated estimates. It’s raw request activity. That alone makes it more reliable than some of these tools out there.

Each AI model has variations of its own bots for different purposes. I recommend using AI to break these down, but it’s worth excluding certain bots from what you look at in this data..

For example, for ChatGPT:

The OAI-SearchBot is the one you want to pay attention to as that’s what will show the potential for citations.

Also pay attention to ChatGPT-User requests. These are active leads. A real person is asking ChatGPT to reach out to your site.

Paths can show you that AI crawlers are accessing certain URLs over and over - or not at all.

Start by looking at the URLs with the most repeated requests. Then review them as a group. These pages have the biggest opportunity to be cited.

Then audit these pages.

If the page is irrelevant, try to be creative and add information to the page that links and guides the user and crawler somewhere better.

Otherwise, 301 redirect it to a relevant page (not the homepage or an unrelated page). This passes the ‘authority’ to the new page and tells the bot exactly where to go next time.

If a page is already attracting repeated crawler interest, improve that page. Make the answer clear. Improve the structure. Add better internal links.

The same process applies to pages that aren’t getting crawled, except you ignore the non-important pages.

Focus on pages that you would want to show in AI search, audit them, update them, and track performance to see if crawlers are accessing.

This could also be down to poor incoming internal links, poor topical authority, poor structure, etc. There are many reasons you would have to audit, diagnose, test, improve, and monitor.

It involves some manual work, but it’s worth it.

Patterns are more important than paths because they show whether the activity is more focused on certain areas of your site.

If you keep seeing activity across something like /guides/ai/, /compare/, or another clear section of the site, that points to topic-level or format-level traction.

It shows that AI systems are not just checking one page, but keep referring to a whole area of the site.

They might see you as a topical authority on a certain subject, or you might have optimized all of those pages compared to the rest.

This provides an opportunity for content.

You should audit and improve the internal links across that section. Then look to expand coverage around the subtopics already getting crawled.

Make the strongest page in that cluster clearly better than the rest.

I recommend using AI to run a content gap analysis here. Get the category or folder, or whatever is being visited most by the crawler, and use AI to browse and scan for missed opportunities.

You can also see whether the wrong section is getting more visits. If repeated activity sits in low-value folders or outdated templates, the site is making those assets easier to find than the ones you actually want used.

Patterns tell you where to invest. Paths tell you which pages you should optimize.

Check which kinds of pages keep appearing across the paths, getting crawler attention. That can tell you a lot about how AI systems are reading your site.

If informational content is accessed repeatedly but commercial pages aren’t, your site may be seen as useful for context, not decisions. Fix that by linking decision-stage pages into the informational assets already getting attention.

If one content type keeps showing up, inspect its structure. The win may come from deeper information, better headings, lists, or tables rather than the topic. Apply those strengths to other important pages.

If crawl activity is high but clicks are weak, that content might be getting used without giving people a reason to visit.

Finally, if activity keeps clustering in one topic area, that’s where the site already has more authority. Use that to support adjacent topics instead of building them in isolation.

You should also scan for patterns in 3xx, 4xx, and 5xx responses.

If you see repeated redirects on high-request paths, that means AI bots are hitting outdated URLs. You’re introducing challenges for them to reach the final page.

Update internal links so the correct URL is the first touchpoint. Clean up redirect chains. Remove unnecessary steps.

If you see 4xx or 5xx errors on pages that attract requests, you’re blocking access to content that already attracts AI. Fix those immediately.

This is one of the few areas where the action is obvious. If access is broken, nothing else matters.

But don’t over-index on this. It’s maintenance, not strategy.

Repeated requests don’t mean citations, traffic, visibility, or revenue.

The logs don’t tell you if the page was used, only that it was considered.

You need to pair this data with engagement and other metrics. Look at referral traffic, time on page, conversion rates, and downstream user activity.

Crawler data tells you where attention might be. It does not confirm value.

Read the full article here.