The Seotistics newsletter is written by Marco Giordano, a Data/Web Analyst with the goal of combining business and web data. Tired of the usual boring Analytics content without any business impact? Seotistics teaches you how to use Analytics, web data and even content in your workflow while helping you with Strategy.
Share
AI Measurement: The Final Issue
Published 5 days agoΒ β’Β 6 min read
β
Use Data Or Be Used By Data!
The July 6 issue of Seotistics is here for you!
This is the last issue talking about LLM measurement.
I will cover some of the last details because this topic has been covered a LOT.
Then, I will start drafting one of the many articles I promised you π
If you are still postponing studying LLMs... I have bad news: they are everywhere.
It's also true that AI Search as of now looks like Sci-Fi in terms of measurement (it doesn't exist yet).
Please move this email to your Primary inbox or reply to it. This is to prevent Seotistics goes into spam by accident. Gmail users can read this tutorial to do it.
Use the code "MARCO" to get 25% on the ticket at checkout, see you there!
The Limitations of Citations & Mentions
The last time I mentioned citations... which are useful overall but could be misleading.
In fact, you can be mentioned without being cited... that is still a win in my book!
auto (dot) com would get the citation (source) and Ferrari is just the mentioned brand.
This is why I recommend looking at both. Results from 3rd parties will naturally include mentions of different brands with the citation (the source) of one website.
Use them as an indication of whether you are picked by LLMs.
Remember, you want to be visible for some core topics you define or people won't choose you.
The Different Models
As mentioned last time, models greatly differ to each other.
AI Mode for example cites a lot because it acts more like a research assistant and a Google extension.
If you've ever used Claude, you'd know it tends to be more conservative with web searches.
You'd tend to use the metrics from the past issue for both, though.
Regardless, the 2 have very different use cases.
Claude (right) didn't run web search, while AI Mode naturally works based on Google.
My advice is to avoid obsessing over details and overoptimization because you will commit a lot of mistakes.
Treat LLMs as "similar" just to avoid overcomplicating your thought process.
P.S. If you remember the Claude Code leaks, the of a page is ignored which means no schema.
This was already obvious sooooo, nothing new.
Attributing Results To AI
You may have heard (or not) about incrementality as a way to measure the real impact of a given channel/source on revenue.
This is NOT easy at all because it requires companies to have a proper setup and system to track it all.
In the proper conditions, you could:
apply very specific changes to a page and measure what happens compared to a page without them
compare between specific regions/areas
compare with user groups
This is close to impossible in reality and most tests we read are Sci-Fi.
The big limit of Organic is that you can't switch it off (unlike Paid).
Well, technically you (partially) could by noindexing your website π
Add updates, all the changes that happen on your website + Google results and you have the perfect recipe for horrible measurement.
In reality this is super hard to replicate and you can't even have holdouts with "GEO".
You can't instruct an LLM to ignore a brand as a test...
Where you can focus today (not many did so far) is your CRM.
It's super beneficial to understand the basics of tools like Salesforce and HubSpot.
Say you work for a B2B company... you use AI and SEO to bring you leads.
Then, you ideally move from leads to actual business opportunities.
Even though a CRM doesn't solve the issue of measuring AI Search, you can measure variations.
β By analyzing your sales pipeline you learn more than what GA4 tells you.
According to the latest research by Similarweb that we analyzed in the 2 previous entries, people run branded searches after AI exposure.
That's a good piece of info that doesn't help us with measuring though.
Measuring direct traffic is also a dumb idea for the reasons I listed in the past.
Content Logic: Explained
The biggest issue with content is that it's still considered homework by companies.
You need to content to rank on search engines and be visible.
Which is correct but it's a horrible mindset to craft art.
This is also why I dislike working with publishers compared to years ago... because now with AI the entire business model is collapsing.
This works very well for Ecommerce. Most content is commodity and often unoriginal.
Drawing the direct connection between content and results is impossible.
Think of content as the mirror of your value proposition.
You do content to articulate your value to people and help them solve issues.
To be perceived as an authority in a field, you need to make sure people remember and cite you.
This is why I write content rather than just sending spammy emails or attending conferences.
What Fails Today
I see 3 main problems behind why companies can't produce good measurement (and therefore decisions).
data storage
processes
people
Data storage is one of my most common topics as this is super needed and easy to do today.
Without data retention, it's impossible to analyze data and make meaningful analysis (check my resources).
The 2nd issue lies in the processes.
Companies barely think about optimizing how they work and often rely on legacy tools.
This is how corporations work, Enterprise Analytics is on a whole different level of complexity.
Still, LLMs now cost peanuts if you don't use Anthropic or OpenAI models.
I recommend everyone to give openrouter a try and test the models inside n8n. Check my latest LinkedIn posts if you feel confused.
I am surprised many of our colleagues don't lock themselves in their room to play with AI.
The opportunities are endless and a lot of legacy processes can be radically changed.
Speaking of which, I recently discussed how Fable 5 (Anthropic's model) is overhyped.
The model has been severely nerfed and costs a LOT.
For comparison, GLM 5.2, a very good Chinese model, costs $2.8/1M token agains the $50 of Fable 5 lol
The final issue is... people!
If SEOs refuse to upskill and analysts refuse to learn marketing/business, organizations will never move forward.
Beyond The Data
So as you have noticed, I myself don't believe much in all these new frameworks and methods used by people.
That's because in industries like SEO but even Web Analytics, you can count on your fingers people with a strong marketing background.
Brand didn't suddenly became important and "content for the users" is not a new thing.
β
The truth is years ago you could get away with it easily because Google wasn't punishing you.
Why even expand your knowledge or test if you can make money with the bare minimum, after all?
I've recently had very positive results in terms of actions/recommendations with tools like Waikay (topic reports).
I confess I didn't try all of the others apart from Semrush AIO (Enterprise).
Even without a tool, LLMs with proper tuning are super good at giving you advice and analyzing content networks.
The next issue will cover a topic you are familiar with from a new lens.
β Can you guess which one?
π₯ Join Our Community
Our Discord community offers a small place where we can talk business and web data.
If you hate all the noise of social media, then this place is for you.
I will start posting more there as we have a forum channel now.
This is the best way to stay updated in real time on Seotistics:
The Seotistics newsletter is written by Marco Giordano, a Data/Web Analyst with the goal of combining business and web data. Tired of the usual boring Analytics content without any business impact? Seotistics teaches you how to use Analytics, web data and even content in your workflow while helping you with Strategy.
Read more from Seotistics - Web Analytics + Business + Strategy
Use Data Or Be Used By Data! The June 29 issue of Seotistics is here for you! The last time we talked about AI measurement and what changed. Turns out a lot changed indeed! I will go deeper on some concepts that the last time weren't covered with due credit. P.S. Next week the last part of this series. Please move this email to your Primary inbox or reply to it. This is to prevent Seotistics goes into spam by accident. Gmail users can read this tutorial to do it. Read this in your browser π£π£...
Use Data Or Be Used By Data! The June 22 issue of Seotistics is here for you! LLM/AI measurement is on the mouth of everyone... but is it actually possible? I don't want to give you the secret to all problems because it doesn't exist. In fact I will show you why most of this is not as easy as you imagine. This is just an intro for the next content. P.S. I will also write a longform article on the topic, this is just the starter. Please move this email to your Primary inbox or reply to it....
Use Data Or Be Used By Data! The June 16 issue of Seotistics is here for you! The more I work in Web, the more I realize the shift that is happening is more real. SEO, PPC, Social media, you name it, it's just a different game now. Yet, people wage their marketing campaigns with outdated tools and tactics. It's always fun to come back to old problems with new perspectives and tools! Please move this email to your Primary inbox or reply to it. This is to prevent Seotistics goes into spam by...