Author: Terry Cane

Although Google has provided us with the occasional breadcrumb over the years, we ultimately just don’t know how its algorithms really work. The majority of what we know about Google is based on observation. As reported by TechRadar Pro, this may soon change with the Digital Services Act, which goes into full force on Jan. 1, 2024. 

Until then, educated guesswork is all we’ve got. Fortunately, that may be enough for at least a brief explanation of how Google’s algorithm indexes content. It also helps that this is one of the few areas where Google has been at least somewhat candid—knowing how to catch the attention of Google’s crawlers doesn’t confer the same sort of advantage as understanding how the search engine evaluates each and every ranking factor, after all. 

So how does Google decide which content to index? 

Per documentation published on Google Search Central, Google indexes pages through automated software bots known as crawlers alongside an algorithm it refers to as Googlebot. The company uses a nonspecific algorithmic process to determine which sites to crawl, how frequently to crawl them, and how many pages it should fetch from each site. Once it discovers a new site, Googlebot simulates page rendering using a recent version of Chrome. 

To use an analogy, Googlebot essentially functions as a central overseer, monitoring the various nodes under its supervision for any changes using an army of digital drones. During this process, new pages may be discovered either through links to a known page or courtesy of web searches. Google further notes that Googlebot does not crawl every page it discovers, and that there are numerous factors that may cause its crawlers to overlook a page: 

• The disallow flag, which indicates that a page should not be crawled.
• The noindex flag, which indicates that a page should not be indexed. 
• A login process that renders the page inaccessible without authentication. 
• Network problems.
• Server issues. 

Although Google’s URL discovery is largely automated, there are two ways you as a website owner can trigger a manual crawl.

The first is to manually build and submit a sitemap to Google to help it crawl and index your page more efficiently. Google will only examine a sitemap the first time you upload it, or if you upload again to notify it of changes. Submitting a sitemap does not guarantee that it will be crawled immediately, and Google advises against repeatedly pinging or uploading the same sitemap. 

Alternatively, you can use Google’s URL Inspection Tool through the Search Console to submit individual pages for crawling or recrawling. You can only do this if you are an owner or full user/administrator. There is a limit to the number of URLs you can upload at any given time, and each page should only be submitted once if unchanged. 

There’s obviously a bit more to Google’s indexing process than we’ve described here. Unfortunately, we aren’t privy to those details, which Google keeps close to the chest. On the plus side, at least you now know a bit more about indexing, and specifically how it plays into your own SEO efforts. 

Wondering how much you should spend on search engine optimization?

You’re not alone. The industry doesn’t exactly have a history of making itself accessible to outsiders, after all. It also doesn’t help that so many SEO agencies don’t do much beyond selling the digital marketing equivalent of snake oil. 

Small wonder, then, that many small and mid-sized businesses don’t even factor SEO into their budget. And those that do usually spend an average of around $500 a month, according to SEO training and link-building specialist Backlink. Per Backlink, the average agency typically costs between $50-$150 an hour. 

Most businesses probably have very little notion of what they’re getting for that money, either. 

Here’s the thing—where SEO is concerned, you very much get what you pay for. If you work with an agency that charges you peanuts, the quality of service they provide will likely be equivalent to that. Similarly, the agencies that charge exorbitant prices typically aren’t worth the cost. 

This background information is all well and good, but it’s not what we’re here to discuss. We’re here to walk you through the basics of planning your SEO budget. The good news is that it’s actually not as complicated as you might expect. 

• Start by looking at your overall marketing budget. How are your funds allocated, and where are you getting the lowest return? Generally, you should be allocating anywhere from 20-40% of your marketing spend to organic traffic. 
• Consider what you need to do. Are you creating a website from scratch and need someone to handle every single facet of optimization, or do you simply need an agency to crawl your site and let you know what you’re doing wrong? 
• Define your goals. Similarly to the above, what exactly do you want to gain from SEO? You want to make sure you set clear, measurable, and attainable objectives—be realistic. 
• Ask what you’re willing (and able) to do on your own. With the proper guidance, SEO isn’t terribly difficult to understand. If you have the time to train yourself with a resource like Moz’s Beginner’s Guide to SEO, it may be worth your time to simply pay for a premium SEO tool and manage things yourself. 
• Look at competitors. How does your site stack up to others in your industry or niche? 
• Use a cost calculator. Websites like SEOcalc should generally be taken with a grain of salt, but they can nevertheless give you a decent idea of where your starting point should be for your budget based on factors like your website’s age and size, your target audience, how well your keywords rank, and so on. 

It’s important to note that you don’t necessarily need to spend anything on SEO. It’s entirely possible to build and maintain a successful website wholly through free tools and utilities. At the same time, your chances of success are far greater with premium software—or better yet, the assistance of an experienced agency. 

One thing many people don’t know about search engine optimization is that it actually consists of two distinct disciplines—on-page SEO and off-page SEO. At first glance, the difference between the two should be fairly obvious. It’s right there in the name, after all. 

At the same time, we still feel it’s important to take a bit of a deeper dive into the core characteristics of each one, if only to help you better understand SEO as a whole. 

On-Page SEO

This is what a lot of people immediately think about when you mention SEO. It’s all the things you do on your own website to make it more attractive to both your audience and Google’s algorithms. The factors that directly impact on-page SEO include:

• Page speed
• Content quality
• Mobile friendliness
• Ease of navigation
• Load time
• Interactivity
• Visual stability
• Alt text, meta titles, and meta descriptions
• URL
• Keywords

Yeah, there’s kind of a lot—but that’s sort of to be expected. On-page is arguably the bread and butter of every SEO strategy. It’s all the stuff that’s directly under your control, meaning you can put as much effort into it as you’d care to. 

Off-Page SEO

You probably know off-page SEO by another name—marketing. It’s basically a catch-all term for everything you do to promote your website and its content across the web (paid advertising aside). Some examples of off-page SEO include:

• Guest posts published on other websites
• Backlinks to your site
• Social media promotion
• Unlinked mentions of your business or brand
• Your Google My Business page
• Management of online reviews

As you’ve probably guessed, you have a bit less control over off-page SEO since it’s all external stuff. That’s not to say there’s no point putting any time or effort into optimization, mind you—quite the contrary. 

Your SEO Strategy Should Include the Best of Both Worlds

On-page SEO and off-page SEO are ultimately two sides of the same coin. A good SEO strategy should make equal use of both. That said, on-page SEO must come first—it’s both the foundation and a springboard for off-page SEO. 

After all, you don’t want to spend a bunch of time and effort promoting a website that’s a usability disaster or has little to nothing in the way of quality content. 

Beyond that, we’ll leave you with one final piece of advice. Don’t obsess too hard over optimizing your site. While SEO is still important for bringing in organic traffic, what’s more important than anything is the kind of content you publish. 

You can still succeed on the search engine results page if you have quality content that you haven’t bothered to optimize. People may still stumble across your site, read the content, and share it. The opposite, however, does not hold true.

As the adage goes, good things come to those who wait.

These days, that saying seems like more of an outmoded cliche than genuine advice. We live in a world built to stoke our impatience, a civilization defined by instant gratification. In such a world, it’s all too easy to mistakenly assume that anything requiring a bit of time and effort isn’t worth doing. 

This very much applies to search engine optimization (SEO). Although the shysters and scam artists that infest the SEO industry would have you believe otherwise, it’s not a strategy that will generate results overnight. It’s focused more on the long-term—some SEO campaigns take months or even years to achieve results. 

SEO is not a set it and forget it strategy, either. It requires constant, ongoing effort and improvement. You must be willing to not only perform regular keyword, audience, and sentiment research but also regularly produce high-quality content. More importantly, you must understand that there is no such thing as guaranteed results in the world of SEO. 

This is because at its core, SEO isn’t solely about traffic. Not really. It’s about getting as many eyes on your website as possible and ensuring everything is in place to capture and hold people’s attention. 

It’s about building authority, establishing a strong reputation, and cultivating relationships with your audience. Unless you’re dealing with someone who’s a walking red flag, none of this happens overnight. These things take time and persistence. 

It’s also important to emphasize just how much content exists online, even within your industry and niche. You have scores of competitors all vying for the same audience you’re targeting. Many of these competitors have likely been doing this for far longer than you have—meaning they have a head start. 

Although it can be tempting to look for shortcuts, we strongly advise against: 

• Regularly changing your domain name. 
• Completely changing your content strategy. 
• Targeting competitors with ‘black hat’ techniques. 
• Trying to buy your way to success. 

If it helps, think of SEO as a sort of digital gardening. Your initial keyword research plants the seeds, but it’s up to you to cultivate them and help them grow into something tangible. And while it’s certainly possible for these things to grow on their own, the best results will come to those with the right blend of patience, persistence, and skill.

So be patient. Learn to navigate the complexities of SEO. Approach your optimization efforts strategically, deliberately, and intelligently.

In the long term, it will all ultimately be worth it. 

How much should you spend on search engine optimization (SEO)? 

That’s a challenging question. One made all the more difficult by the fact that no two companies will have the same answer. There are many different factors at play where SEO is concerned. We’ll go over some of the most prominent.

Here are five things you must account for when planning your SEO budget. 

Your Current Website

Are you starting fresh and optimizing for an entirely new website, or are you looking to drive traffic to an established brand? Have you put any thought into your website’s information architecture, content quality, and backlinks? Are you currently suffering any algorithm penalties?

These are all questions you need to ask yourself at the outset before you even begin planning a strategy. 

Objectives

What do you want to achieve with SEO? Measurable, realistic goals and milestones will help you track your progress and help you figure out a baseline for how much you should spend. The keyword here is realistic. 

Avoid striving for a specific place on the SERP or trying to generate explosive traffic in just a month. You need to understand that SEO isn’t immediate. It takes time to get results. 

Once you’ve established your goal, try to estimate how much additional revenue your website will generate once you achieve it—that figure can then be used to guide how you spend. 

Marketing Budget and Spending Limits

In most cases, SEO is not going to be the only line item on your marketing budget. You’ll likely have to balance it with things like paid social, pay-per-click (PPC) advertising, and inbound marketing. To figure out how things should be allocated, you need only ask yourself a simple question.

In a perfect world, what’s the maximum return each of these investments will generate? 

Current Traffic and Conversions

While a small or mid-market business certainly could dedicate its budget towards enterprise-level SEO, that doesn’t mean it should. Just as the current state of your website represents an important SEO starting point, so too does your brand. Consider the following when budgeting: 

• Monthly traffic numbers.
• Conversion rate.
• Average order value per customer, if relevant. 
• Other marketing channels. 

Your Competitors

Last but certainly not least, look at what your competitors are doing. If you can find and assess a business that’s similar in size to your own, you’ll be able to determine your SEO budget more accurately. More importantly, this research can help you identify potential opportunities—weaknesses in a competitor’s brand, for instance, or a gap your business could potentially fill. 

The Right Budget Doesn’t Break the Bank

Taking into account all of the above, we’ll wrap things up with one final piece of advice. The right budget is one that you can comfortably afford while still generating a reasonable return. Keep that in mind, and everything else should easily fall into place. 

This story should be familiar to most of you.

You’ve come up with an excellent idea for content—something you’re confident will resonate with your audience and bring in qualified leads. However, when it comes time to start researching keyword permutations for that topic, you’re met with a rather unpleasant surprise. No matter how you phrase or rephrase your terms, no one seems to be searching for the topic. 

Back to the drawing board, right? 

Maybe not. As it turns out, low-volume keywords can be just as valuable as low-hanging fruit, if not more so. As noted by marketing expert Neil Patel, this comes down to a few factors:

• Clear intent. This means that any content to which the search phrase is connected will be highly relevant. 
• Low difficulty score. As you may already know, the lower a keyword’s difficulty score, the easier it is to rank for that keyword. 
• Length. A long-tail keyword that contains a secondary, more valuable or higher-volume keyword may not appear worth targeting on the surface. However, it can still generate considerable traffic—and again, there’s the matter of intent. 
• A highly niche topic. Some topics are going to be low-traffic no matter what you do—for example, a blog that exclusively targets business brokers and their clients may appear to primarily contain low-volume keywords. But the people who search for those keywords have high intent. 
• Low cost per click. A keyword’s cost per click is another indicator of its competitiveness. Lower means the keyword has less competition, meaning it’s easier to rank for it. 

There’s also one more factor to consider—your competition. Competitors are very likely taking the same approach as everyone else, targeting relevant keywords with a reasonable search volume and difficulty score. 

By switching your focus to lower-volume keywords, you might well be able to get the jump on them from a marketing perspective. As long as you understand your audience, what they want, and what they search for, low-volume and non-competitive keywords can be just as valuable as high-traffic keywords. In some cases, more so. 

“Ignore the metrics that everyone else is using to select their keywords,” advises SEO expert Dmitri Dragilev. “Instead, focus on keywords where you can intercept the customer in the middle of the purchase decision, piggyback on the authority of an established player in a related field, [or] offer a better solution to a problem posed by an established player in your field.” 

At the end of the day, targeting low-volume keywords is really a matter of quality over quantity. After all, which would you rather have for your business? 

• Content that brings in 50 qualified leads but generates no additional traffic. 
• Content that brings in thousands of leads, but none of them convert. 

It’s simple mathematics at that point, really.