Use Data Or Be Used By Data!

The July 13 issue of Seotistics is here for you!

Scraping is one of the most important skills you must have as an analyst AND marketer.

But what changed after LLMs? Well, a lot!

This issue is about YOU scraping websites to get data, not how LLM providers scrape your website.

Please move this email to your Primary inbox or reply to it. This is to prevent Seotistics goes into spam by accident. Gmail users can read this tutorial to do it.

Read this in your browser

📣📣 Important Announcements 📣📣

Seotistics Summer Sales will soon be available, so prepare your wallets in advance 👀

Including my new course below, so have a look now:

Become a Top Analyst!

I am also working on new updates and content as we speak.

The summer period is busy and my laptop is melting but don't worry, everything will be done!

The Legacy Methods

Most developers are still using outdated methods to carry out tasks that would be trivial today.

Analysts should be "smarter" on this regard and avoid doing extra work.

The main issue with scraping is DOM changes, meaning that if there is a redesign or a small component is changed, your script will break.

If you learned Python like many people you should be familiar with:

requests
BeautifulSoup
Selenium

This is in the 101 class you attended years ago and still in most tutorials today.

If you've been following Seotistics, you already know most of this stuff is either outdated or unneeded 👀

The New Meta

Thanks to LLMs and well, more effort in general, now the scraping meta has radically changed.

I've found myself more and more comfortable with new libraries such as Crawl4AI, Scrapling and even ScrapeGraphAI.

Scrapling also has MCP support which is amazing if you consider you can embed it into a Claude skill!

The other thing is that before HTML was OK as output and this is what I was also doing before.

Now with LLMs, the output is better off as either .json or .md for token efficiency and understanding.

So HTML as the final output is outdated folks.

Scrapegraphai is also another great library most of you will like since it takes prompts!

This is great for irregular domains and web pages, kinda like Crawl4AI but... it can quickly stack up.

To replace the usual "requests" library, try curl_cffi, it's the same but better!

But Screaming Frog!?

Screaming Frog is one of the best tools out there but for crawling websites and do technical checks...

which is quite different from scraping.

The good thing is that since some time it supports a feature to manually select what you want to extract from a website.

Example of me using Screaming Frog to extract the text of the CTA.

This option is available from Configuration -> Custom -> Custom Extraction and then adding a new extractor and clicking on the grey browser icon.

Screaming Frog also has scraping capabilities as you can see.

Competitive scraping requires more though, like customization.

Use Cases

The use cases didn't really change but now you can scale beyond the mainstream:

Preparing content for AI/LLMs
Better scraping in general
Large-scale scraping

You still need to be technical and spend a LOT of time on those (still less than before).

I am working on all these use cases and I can tell you there are many blockades and obstacles.

What I scrape I save into a qmd collection... what is that? Check my resources :3

This is why you should ALWAYS start with the problem and then develop a solution.

For Seotistics, I just need to process my content and that of some competitors...

doesn't really matter to do gigantic scrapes.

The opposite can be true for clients or complex projects... where you need to scrape entire websites 👀

Preparing Content

As shown multiple times, this is my favorite use case.

No sane person would ever do content research manually in 2026!

This is why it's important to have something in place, which often involves retrieving information online.

A simple workflow you can try today is the following:

As said before, n8n + openrouter is a great combo to start with.

DataForSEO (or any alternative) is my golden standard to start research.

By now, you should be quite familiar with it since it's simple to use and well documented.

Then, you should get a list of websites and potential competitors, based on the input you provided.

You can explore my repository to see how I do it.

From there, you scrape what you need and use LLMs to summarize the results.

Passing a prompt in n8n (but you can also use fileserver, go check my latest LinkedIn posts).

This is where it gets fun and original and you have to spend the majority of your time.

n8n has usual has great support for DataForSEO!

What the majority of the people get wrong is that they think content is copying what others do and adding more details.

This used to work before but now requires more effort.

In B2B there is no such a thing as copying from others because you would get crushed.

What often misses is the authority and distribution of the content.

Scraping Examples

From a pure code perspective, scraping did change because now you can write impossible code with LLMs.

Years ago, it would have taken ages for you to develop some complex scraping script.

You can generate stuff like this much more easily today.

Now it's faster to have an MVP you can test and fix over time.

If you are a budding Analyst or a marketer who needs some projects, scraping is for you:

Ecommerce Monitoring (scrape prices and assess variations)
Sentiment Analysis (can be done on any text, really)
SERP analysis (once again, already covered by me and you can build on top of it)

I know what you are gonna say "but Marco now with AI everyone can do it!".

Not true, most people are lazy and don't put any effort, so this is your chance to be different.

With practice you also get to learn actual problems that happen in real life, like getting blocked by their firewall.

Large-Scale Scraping

For more complex use cases like creating your personal aggregator, you need to be quite smart about your project.

This is the field of actual software engineers and not analysts... but there are cases where you can find workarounds.

use curl_cffi for faking your bot as a real user
Playwright to handle Javascript-heavy websites (or just use the 3 libraries I mentioned above lol)
Proxy rotation which means you send requests from multiple IP addresses
Storage -> doesn't have to be BigQuery here, actually the opposite

In most cases, you won't deal with such complex scenarios so all good. 👌

And it's even better to use APIs and avoid scraping altogether, if possible.

We know it's often not possible or convenient, so we scrape 👀