Keyword Competition Research
- Keyword Competition Research
- Why Is This Important?
- Techniques for Identifying Keywords
- Keyword Competition Research
- Keyword Competition Research - Process
This guide assumes that you are familiar with keywords, the concepts, terminology and metrics that describe them.
There are many techniques and types of keyword research that can each play a different or complimentary role in the ongoing endeavour to find new valuable keyword. We cover a range of common techniques for performing keyword research in this and other guides in this series.
This guide looks at diving into the detail of how competitive a group of keywords are, and what it takes to rank well for them.
Choosing the right keywords or the best keywords and having this rolled up into a keyword strategy is pivotal for any site. There is no point spending time, money and effort targeting keywords that no-one searches for or that you can’t rank for. In some cases, small tweaks to the target keyword can lead to dramatic changes in the volume or quality of traffic.
This type of analysis is important for understanding a much more specific set of goals and targets for where your site needs to be to effectively rank for keywords. This guide and the processes detailed in it enable you to understand your keywords in more depth. You can also use this technique to refine a keyword list or identify new keywords.
We cover the long-term strategy and how keyword research feeds into this in another guide on keyword strategy.
We have mentioned in several more basic guides the concept of competition or difficulty rank for a keyword… Metric that are used to describe this can provide a valuable top-level insight into where you might be abler to rank and for what. This type of analysis dives deeper, looking at the components that make something difficult or competitive.
We look at backlinks, content and a range of metrics to measure those, in order to understand what we need to achieve in these areas to effectively compete. We touch on this technique in our guide to competitor backlink analysis and the fundamentals of both backlinks and content are covered in these guides.
There are many avenues to finding and ultimately finding and targeting keywords... There are also many factors in determining what to target and when to target it, often this is a balance between cost and pay-off, how much effort do you need to put in to get more out?
There are no one-size-fits-all solutions, and the options available to your will limit the processes that you must go through.
If you have a list of keywords that you want to understand better than you do, you might pull competitor data in for each keyword. You will require access to a range of tools and data to perform this type of analysis and due to its more advanced nature we are assuming a certain level of knowledge already.
This is a much more strategic approach to keyword research and is not the sort of larger scale high keyword volume research we discuss in our guide to basic keyword research.
How you choose to represent or analyse this will vary from person to person; but typically, you will use this analysis to find opportunities or understand barriers to entry for specific keywords. Feel free to adapt this process, swap out, remove or add datasets and metric, to suit your needs.
We won’t spend too much time on this, you can acquire keywords in any number of ways; from identifying them in a larger keyword research project to a client handing them to you and saying how can we rank for them. Any more than a hundred keywords would create a lot of data and take a lot of work to get and analyse…
We will assume for the purpose of this, that you have a set of keywords you want to look at in more detail.
These are defined as part of the process, once you have a group of keywords the aim is to find out who ranks for them. We most commonly use the top 20-30 unique organic ranking URLs for each search, but you could get by with the top 10 if you were limited for any reason.
You will need to crawl the SERPs (Search Engine Result Pages) and pull out all of the ranking URLs in the top X number of organic positions. For each keyword on your list, you will need to do the following SERP data:
- Record each ranking URL
- Record where each URL ranks organically
- Record the ranking domain if different to the ranking URL
- Organic listing type
There’s a huge range of data that you can pull in to do this analysis, this falls into two main categories:
We would typically pull in the following off-page data about each ranking URL (& root domain), this allows us to see the backlink topology of the SERPs for our chosen keyword:
- Number of backlinks (for ranking URL and root domain)
- Number of referring domains (for ranking URL and root domain)
- Number of referring IPs (for ranking URL and root domain)
- Number of .gov & .edu linking domains (for ranking URL and root domain)
- Number of exact match linking domains (for ranking URL and root domain)
- Keyword usage in anchor text (%)
- Trust metric (e.g. Trust Flow) for backlinks
- Authority metric (e.g. PA & DA) for backlinks
To be clear by ranking URL and root domain, we mean that if the following URL ranked organically for a keyword:
In the above example, the above URL would be the ranking URL, & example.com is the root domain.
This data helps you to create a benchmark for a keyword, you can more often easily see in a table or a chart where your page (targeting the keyword) sits in this spectrum. This can provide an insight into how well you could expect to compete and rank in this area; but also provide targets to help you improve your backlink profile where it is required to more effectively rank.
Thus, this type of analysis can provide some strong strategic input into your link building targets.
The following list is the on-page data that we would pull in for each of the domains in our list:
- Word count of ranking URL
- Keyword density of ranking URL
- SEO Score for ranking URL (Raptor metric)
- Number of pages on the ranking domain
- Average no. of words per page on the ranking domain
- Total number of words on the ranking domain
- Keyword usage throughout ranking domain
This data allows to get an idea of how much content a site has, how well optimised that content is for the keyword in question. Again, plot this in a table and charts to see where your site sits on the spectrum.
Other Checks / Data
You can also start to derive more data from the existing data, such as:
- Is keyword in ranking domain?
- Is keyword in ranking URL?
*It’s worth considering that if you pull in more than one metric to represent a single concept like UR (URL Rating) and PA (Page Authority) … You then have to weight these scores or you risk unbalancing the equation should you choose to apply a formula to it.
Sometimes it’s better to limit the data to a smaller set to avoid ‘analysis paralysis’, there are a large number of metrics, ratios, measures and data that you could look at… The only way to approach this is to include what you believe to be most influential and most actionable.
You could for example, pull in social media data into this matrix, some brand equity score or whatever you want!
This process is fairly well known among SEOs, the idea is that you create categories and within them, labels… The example data below shows what we mean by this:
In this example, we show some keywords relevant to us with two categories, the first ‘product’ has multiple labels (SEO, Backlink, Keyword, etc). The second category has only two labels, which are either ‘yes’ or ‘no’, these delineate whether the keyword refers to a tool (program, software, app, etc) …
You could structure either in any way you want, the important thing is that you are consistent in your application of the labels.
Add as many categories as you need to properly categorise your keyword data. We provide a tool for this, or you can use a spreadsheet (download one from here). This will allow you to very quickly and easily create categories and labels for your keywords; as it automatically applied the specified labels to keywords containing the specified text.
As you progress, filter on categories to look for keywords that have been assigned no category, these are usually marmite… Because these keywords are ones that you hadn’t anticipated, they are often the most interesting or the least relevant!
You need to spend a little time thinking about you will address the structure of the categories, a poor implementation initially can lead to redoing work or an illogical output. One advantage of this technique is the meta-analysis that it allows us to perform on this data by segmenting it by category or label… Read more about this is our guide to keyword data analysis.
There are several techniques that we would typically implement when analysing this data in order to derive actionable insights and inform our strategic decisions. We have touched on some of these already, but expand on these concepts below.
Seeing where your site sits in either the spectrum of backlinks or content allows you to see where you need to make improvements. If your site or page has no backlink and all the ranking URLs have at least 20, with the top being 100… You have your work cut out for you, but you do have some clear achievable targets.
You also need to look at how well you compete across all of the different areas, if you already have a site with 10million words and 20,000 pages; even if you are a few million behind the competitor outranking you… You would need a huge effort to make a small impact, adding a million words could take a year and provide a 10% increase.
Continuing this example, if the backlink profile was quite weak by comparison to the competition and volumes quite low… It might be that your time is better being weighted more towards link acquisition than content creation, because you can make a bigger difference in that area.
Considering the amount of effort involved is also a crucial component of this analysis, the amount of work required might be a prohibiting factor or barrier to entry for your business. Consequently, a set of keywords might be unattainable in the timeframe of your current strategy.
Another factor to consider is the composition of the SERPs… Are local listings at the top of the page, is it mostly Wikipedia and educational sites listed, are there many businesses, etc?
Understanding this can provide some insight into the most common intent behind a keyword and in turn this can inform your strategy. If it’s all local listings, map listings, gumtree and yell… Your strategy might be better geared towards creating, optimising and promoting this type of off-page listing.
There are really two components to this;
- The listing type (knowledge graph, map, etc)
- The type of site listed
You need to understand both to understand what you need to do and if it’s possible.
Remove Anomalous Data
We typically recommend looking at the data both with and without the anomalous component for completeness. This is best explained with an example; if Wikipedia is ranked in the SERPs alongside a bunch of local businesses, it is completely reasonable to remove it from your analysis when calculating averages for example.
If one site has 150million backlinks and all the others have less than 1,000 you need to remove it, to have any useful data.
That is not to say you ignore the data completely… if the top three sites are all huge mega-brands, you will need to accept that fact and factor it when deciding what keywords to target; you just don’t need to include their data in charts and averages… Or you have multiple benchmarks for groups of positions (1-3, 4-10, etc).
Categories & Labels
You can use the categories and labels that you assigned to the keywords in order to segment and analyse the data you have been collecting. Using pivot tables, you should be able to quickly pivot the data to see things like:
- Average authority per keyword category
- Average backlinks for the ranking URL per keyword category
- Keyword categories with the most words per ranking URL
You can probably easily see that this sort of data provides some really nice meta-analysis about the keywords being researched. This is more useful if there are a significant number of keywords in each category or label… Categories of one are neither good nor useful!
The competition and keyword difficulty metrics are derived from an analysis of the factors we are looking at in this analysis… You can always try to create your own formula to score this, but in reality, it doesn’t provide a huge amount of additional value. These scores already exist, our aim in doing this type of analysis is to dig deeper and really analyse the components of that score.
If you are dealing with vast amounts of data, a formula that derives a score can be a useful way to help guide your analysis, but shouldn’t ever be considered the output of it.
It is well worth looking at a few sites that rank well more closely… Go to the ranking pages, poke around the site, try to get a feel and understanding of their strategy. There are so many nuances that can’t be pulled out by a crawler, the tone of voice, how the content engages with the audience, its structure, how well its supported by other media.
This should provide some insights into how your competition are targeting keywords, are they service pages or blog posts that are ranking, do they cluster content in groups of contextually relevant subjects, etc?
6. Derive Keywords or Make Decisions
At this point you should have a nice clear picture of where you sit in the competitive spectrum across a range keywords and categories… The only thing left to do is to let the data and your analysis of it guide the outcome… Maybe you drop some keywords, decide to go all out for others, set targets for content and link building.
The outcomes of this process will differ for every business, there are no cookies cutter answers, let the data guide and your objectives temper your decisions.