How We’re Automating SEO with BigQuery

The SEO industry is facing a data-scale problem.

There’s so much SEO data that needs to be reviewed in order to get top results for clients.

Keywords, links, competitors, SERP data, technical items, content items…the list goes on.

This data lives in half a dozen platforms – it takes hours just to pull and format the data.

As agencies, we’re forced to account for those hours by raising retainer prices, making quality SEO services unaffordable to most businesses.

Last year, I joined WEBRIS as Chief Automation Officer to change all of this.

My goal was simple:

Automate the busywork, not the analysis.

At the end of the day, SEO is still a service that requires trained human brains. We wanted to free those brains to focus on actions, not Spreadsheet busywork.

I came up with a solution using Google BigQuery.

In this article, I’m going to show you how BigQuery has helped us scale through automation.

WTF is Google BigQuery?

Here’s how Google explains it:

“BigQuery is a cloud-based data warehouse for storing + processing large datasets.”

Here’s how we explain it:

Allow me to illustrate with an example.

Let’s say we’re working on an eCommerce shop with 20,000 SKUs.

Pulling the data down from each tool, formatting, analyzing…20+ hours of busywork.

BigQuery accepts all that data (via API) to process and format at lightning speed. No more slow browsers or crashed spreadsheets, a few clicks and you’re ready to analyze.

But wait…there’s more!

We can sync our BigQuery database with numerous visualization tools:

• Google Sheets
• Google Data Studio
• Tableau
• Looker

What’s that mean? We’re able to combine data from different platforms into powerful, custom analysis reports.

We’re able to get completed audits, ready for review, within a matter of minutes.

Examples of BigQuery in action

We’ve been able to push all of our deliverables into BigQuery. We call each of these deliverables a “Recipe” – Recipes automate the tedious, repeatable reports that our team runs every day, and allow us powerful customization on a per-project basis.

Recipe example: The Website Quality Audit (WQA)

This Website Quality Audit is more than just a standalone Recipe. We call this the “swiss army knife” of SEO audits because it can be used in so many ways. Part technical audit, part content audit, part keyword research tool, it does a little bit of everything.

This allows us to dive into our next campaign steps – keyword research and content creation – with a complete roadmap.

What goes into the Recipe?

• A full Deepcrawl site scrape
• Google Analytics traffic and conversions
• Google Search Console impressions and CTRs
• Majestic links
• SEMrush keywords

What do we automate in BigQuery?

• Map URL level data from all sources (keyword rank, sessions, backlinks, etc.,) to create an SEO data ‘map’
• Auto-assign page-level actions, based on a set of SEO best practice rules
• Auto-assign a ‘page type’ to each URL (i.e. blog post, product page, etc)

Imagine you have a 10,000 page eCommerce site to work through, how are you supposed to make a content plan for each page? No one could categorize those pages effectively, in a reasonable time frame, so we automate the organization so we can focus our efforts where it matters most.

How to we analyze the Recipe?

The output is a Google Sheets audit (full of all the SEO data in the world) and a Data Studio dashboard. We’ll push this audit forward into every other phase of our SEO Sprints.

• Read our full post on The Website Quality Audit

Want to Work With Us?

Because BigQuery makes pulling this data easy, our team uses these reports and more on a daily basis with clients.

This efficiency frees us up to put more focus toward the execution of our unique service offerings – SEO Sprints.