How Does the Google Algorithm Index Content?

Although Google has provided us with the occasional breadcrumb over the years, we ultimately just don’t know how its algorithms really work. The majority of what we know about Google is based on observation. As reported by TechRadar Pro, this may soon change with the Digital Services Act, which goes into full force on Jan. 1, 2024

Until then, educated guesswork is all we’ve got. Fortunately, that may be enough for at least a brief explanation of how Google’s algorithm indexes content. It also helps that this is one of the few areas where Google has been at least somewhat candid—knowing how to catch the attention of Google’s crawlers doesn’t confer the same sort of advantage as understanding how the search engine evaluates each and every ranking factor, after all. 

So how does Google decide which content to index? 

Per documentation published on Google Search Central, Google indexes pages through automated software bots known as crawlers alongside an algorithm it refers to as Googlebot. The company uses a nonspecific algorithmic process to determine which sites to crawl, how frequently to crawl them, and how many pages it should fetch from each site. Once it discovers a new site, Googlebot simulates page rendering using a recent version of Chrome. 

To use an analogy, Googlebot essentially functions as a central overseer, monitoring the various nodes under its supervision for any changes using an army of digital drones. During this process, new pages may be discovered either through links to a known page or courtesy of web searches. Google further notes that Googlebot does not crawl every page it discovers, and that there are numerous factors that may cause its crawlers to overlook a page: 

  • The disallow flag, which indicates that a page should not be crawled.
  • The noindex flag, which indicates that a page should not be indexed. 
  • A login process that renders the page inaccessible without authentication. 
  • Network problems.
  • Server issues. 

Although Google’s URL discovery is largely automated, there are two ways you as a website owner can trigger a manual crawl.

The first is to manually build and submit a sitemap to Google to help it crawl and index your page more efficiently. Google will only examine a sitemap the first time you upload it, or if you upload again to notify it of changes. Submitting a sitemap does not guarantee that it will be crawled immediately, and Google advises against repeatedly pinging or uploading the same sitemap. 

Alternatively, you can use Google’s URL Inspection Tool through the Search Console to submit individual pages for crawling or recrawling. You can only do this if you are an owner or full user/administrator. There is a limit to the number of URLs you can upload at any given time, and each page should only be submitted once if unchanged. 

There’s obviously a bit more to Google’s indexing process than we’ve described here. Unfortunately, we aren’t privy to those details, which Google keeps close to the chest. On the plus side, at least you now know a bit more about indexing, and specifically how it plays into your own SEO efforts. 

How to Protect Your Website from Negative SEO Attacks

The early days of search engines were reminiscent of the wild west. Underhanded or downright malicious search engine optimization (SEO) was commonplace, and many of the top spots on the search engine results page were taken up by low-quality, misleading spam sites. It didn’t take long for Google to correct the issue, and it’s been leveling increasingly harsh penalties against the tactics used by spammers and bad actors, collectively known as black hat SEO. 

Unfortunately, black hat practitioners appear to have missed the memo. Negative SEO attacks are on the rise, driven as much by unscrupulous site owners as by cybercriminals. Today, we’re going to tell you how to protect yourself against them.  

What is Negative SEO?

Negative SEO refers to a specific branch of black hat SEO that involves targeting other websites rather than attempting to improve one’s own page rank. Although the motivations may differ, the end goal of negative SEO is to sabotage another company’s SEO efforts. In some cases, a bad actor might even attempt to directly hack or compromise a website. 

What are the Most Common Types of Negative SEO Attack? 

Common negative SEO techniques include: 

  • Directly hacking a website. 
  • Using link farms or public blog networks to drown a competitor’s site in toxic backlinks. 
  • Content scraping. 
  • Fake reviews. 
  • Fraudulent backlink removal requests. 
  • Fraudulent DMCA takedowns.  

How Do You Stop a Negative SEO Attack?

The short answer is that it depends on the type of attack. The long answer is that in some cases, it’s difficult to know for certain if your website is even being targeted by negative SEO. Some webmasters are quick to blame external factors for their declining page rank, which may cause them to overlook their own mistakes. 

Before you assume you’re being targeted, ask yourself the following questions:

  • Has Google recently updated its algorithm? 
  • When did you last perform a backlink audit?
  • When did you last test your website with PageSpeed Insights
  • Are there any recent industry changes that could explain this? 
  • Has a competitor started outperforming you because they simply have a higher-quality, better-optimized site?
  • Does your website use HTTPS? 
  • Is your website optimized for mobile devices?  

Once you’ve ruled out the factors above, the following steps can help keep you safe from a negative SEO attack—even one that’s in progress. 

  • Check your backlink profile for toxic links using a tool like Ahrefs.
  • Disavow each unnatural link you uncover.
  • Leverage a tool like Copyleaks to manually search for stolen or scraped content, or a platform like DataDome for automated protection. 
  • Every time you see a negative review about your business, make an effort to reach out to the reviewer and rectify the situation, reporting any obviously fake reviews. 
  • Enable spam protection and multifactor authentication on your website, and ensure that you have a secure username and password. 


At the end of the day, most negative SEO practitioners are doing it because they’re either too lazy or too talentless to achieve genuine success; they think they can get away with taking a shortcut instead. The reality is that as Google’s algorithms continue to improve, the efficacy of negative SEO continues to decrease. 

For the most part, as long as your SEO is up to par, you should be fine. 

A Quick Guide to Planning Your SEO Budget

Wondering how much you should spend on search engine optimization?

You’re not alone. The industry doesn’t exactly have a history of making itself accessible to outsiders, after all. It also doesn’t help that so many SEO agencies don’t do much beyond selling the digital marketing equivalent of snake oil. 

Small wonder, then, that many small and mid-sized businesses don’t even factor SEO into their budget. And those that do usually spend an average of around $500 a month, according to SEO training and link-building specialist Backlink. Per Backlink, the average agency typically costs between $50-$150 an hour. 

Most businesses probably have very little notion of what they’re getting for that money, either. 

Here’s the thing—where SEO is concerned, you very much get what you pay for. If you work with an agency that charges you peanuts, the quality of service they provide will likely be equivalent to that. Similarly, the agencies that charge exorbitant prices typically aren’t worth the cost. 

This background information is all well and good, but it’s not what we’re here to discuss. We’re here to walk you through the basics of planning your SEO budget. The good news is that it’s actually not as complicated as you might expect. 

  • Start by looking at your overall marketing budget. How are your funds allocated, and where are you getting the lowest return? Generally, you should be allocating anywhere from 20-40% of your marketing spend to organic traffic. 
  • Consider what you need to do. Are you creating a website from scratch and need someone to handle every single facet of optimization, or do you simply need an agency to crawl your site and let you know what you’re doing wrong? 
  • Define your goals. Similarly to the above, what exactly do you want to gain from SEO? You want to make sure you set clear, measurable, and attainable objectives—be realistic. 
  • Ask what you’re willing (and able) to do on your own. With the proper guidance, SEO isn’t terribly difficult to understand. If you have the time to train yourself with a resource like Moz’s Beginner’s Guide to SEO, it may be worth your time to simply pay for a premium SEO tool and manage things yourself. 
  • Look at competitors. How does your site stack up to others in your industry or niche? 
  • Use a cost calculator. Websites like SEOcalc should generally be taken with a grain of salt, but they can nevertheless give you a decent idea of where your starting point should be for your budget based on factors like your website’s age and size, your target audience, how well your keywords rank, and so on. 

It’s important to note that you don’t necessarily need to spend anything on SEO. It’s entirely possible to build and maintain a successful website wholly through free tools and utilities. At the same time, your chances of success are far greater with premium software—or better yet, the assistance of an experienced agency. 

What’s the Difference Between On-Page SEO and Off-Page SEO?

One thing many people don’t know about search engine optimization is that it actually consists of two distinct disciplines—on-page SEO and off-page SEO. At first glance, the difference between the two should be fairly obvious. It’s right there in the name, after all. 

At the same time, we still feel it’s important to take a bit of a deeper dive into the core characteristics of each one, if only to help you better understand SEO as a whole. 

On-Page SEO

This is what a lot of people immediately think about when you mention SEO. It’s all the things you do on your own website to make it more attractive to both your audience and Google’s algorithms. The factors that directly impact on-page SEO include:

  • Page speed
  • Content quality
  • Mobile friendliness
  • Ease of navigation
  • Load time
  • Interactivity
  • Visual stability
  • Alt text, meta titles, and meta descriptions
  • URL
  • Keywords

Yeah, there’s kind of a lot—but that’s sort of to be expected. On-page is arguably the bread and butter of every SEO strategy. It’s all the stuff that’s directly under your control, meaning you can put as much effort into it as you’d care to. 

Off-Page SEO

You probably know off-page SEO by another name—marketing. It’s basically a catch-all term for everything you do to promote your website and its content across the web (paid advertising aside). Some examples of off-page SEO include:

  • Guest posts published on other websites
  • Backlinks to your site
  • Social media promotion
  • Unlinked mentions of your business or brand
  • Your Google My Business page
  • Management of online reviews

As you’ve probably guessed, you have a bit less control over off-page SEO since it’s all external stuff. That’s not to say there’s no point putting any time or effort into optimization, mind you—quite the contrary. 

Your SEO Strategy Should Include the Best of Both Worlds

On-page SEO and off-page SEO are ultimately two sides of the same coin. A good SEO strategy should make equal use of both. That said, on-page SEO must come first—it’s both the foundation and a springboard for off-page SEO. 

After all, you don’t want to spend a bunch of time and effort promoting a website that’s a usability disaster or has little to nothing in the way of quality content. 

Beyond that, we’ll leave you with one final piece of advice. Don’t obsess too hard over optimizing your site. While SEO is still important for bringing in organic traffic, what’s more important than anything is the kind of content you publish. 

You can still succeed on the search engine results page if you have quality content that you haven’t bothered to optimize. People may still stumble across your site, read the content, and share it. The opposite, however, does not hold true.

Why Patience is the Most Important Virtue in SEO

As the adage goes, good things come to those who wait.

These days, that saying seems like more of an outmoded cliche than genuine advice. We live in a world built to stoke our impatience, a civilization defined by instant gratification. In such a world, it’s all too easy to mistakenly assume that anything requiring a bit of time and effort isn’t worth doing. 

This very much applies to search engine optimization (SEO). Although the shysters and scam artists that infest the SEO industry would have you believe otherwise, it’s not a strategy that will generate results overnight. It’s focused more on the long-term—some SEO campaigns take months or even years to achieve results. 

SEO is not a set it and forget it strategy, either. It requires constant, ongoing effort and improvement. You must be willing to not only perform regular keyword, audience, and sentiment research but also regularly produce high-quality content. More importantly, you must understand that there is no such thing as guaranteed results in the world of SEO. 

This is because at its core, SEO isn’t solely about traffic. Not really. It’s about getting as many eyes on your website as possible and ensuring everything is in place to capture and hold people’s attention. 

It’s about building authority, establishing a strong reputation, and cultivating relationships with your audience. Unless you’re dealing with someone who’s a walking red flag, none of this happens overnight. These things take time and persistence. 

It’s also important to emphasize just how much content exists online, even within your industry and niche. You have scores of competitors all vying for the same audience you’re targeting. Many of these competitors have likely been doing this for far longer than you have—meaning they have a head start. 

Although it can be tempting to look for shortcuts, we strongly advise against: 

  • Regularly changing your domain name. 
  • Completely changing your content strategy. 
  • Targeting competitors with ‘black hat’ techniques. 
  • Trying to buy your way to success. 

If it helps, think of SEO as a sort of digital gardening. Your initial keyword research plants the seeds, but it’s up to you to cultivate them and help them grow into something tangible. And while it’s certainly possible for these things to grow on their own, the best results will come to those with the right blend of patience, persistence, and skill.

So be patient. Learn to navigate the complexities of SEO. Approach your optimization efforts strategically, deliberately, and intelligently.

In the long term, it will all ultimately be worth it. 

5 Things to Account For When Planning Your SEO Budget

How much should you spend on search engine optimization (SEO)? 

That’s a challenging question. One made all the more difficult by the fact that no two companies will have the same answer. There are many different factors at play where SEO is concerned. We’ll go over some of the most prominent.

Here are five things you must account for when planning your SEO budget. 

Your Current Website

Are you starting fresh and optimizing for an entirely new website, or are you looking to drive traffic to an established brand? Have you put any thought into your website’s information architecture, content quality, and backlinks? Are you currently suffering any algorithm penalties?

These are all questions you need to ask yourself at the outset before you even begin planning a strategy. 


What do you want to achieve with SEO? Measurable, realistic goals and milestones will help you track your progress and help you figure out a baseline for how much you should spend. The keyword here is realistic. 

Avoid striving for a specific place on the SERP or trying to generate explosive traffic in just a month. You need to understand that SEO isn’t immediate. It takes time to get results. 

Once you’ve established your goal, try to estimate how much additional revenue your website will generate once you achieve it—that figure can then be used to guide how you spend. 

Marketing Budget and Spending Limits

In most cases, SEO is not going to be the only line item on your marketing budget. You’ll likely have to balance it with things like paid social, pay-per-click (PPC) advertising, and inbound marketing. To figure out how things should be allocated, you need only ask yourself a simple question.

In a perfect world, what’s the maximum return each of these investments will generate? 

Current Traffic and Conversions

While a small or mid-market business certainly could dedicate its budget towards enterprise-level SEO, that doesn’t mean it should. Just as the current state of your website represents an important SEO starting point, so too does your brand. Consider the following when budgeting: 

  • Monthly traffic numbers.
  • Conversion rate.
  • Average order value per customer, if relevant. 
  • Other marketing channels. 

Your Competitors

Last but certainly not least, look at what your competitors are doing. If you can find and assess a business that’s similar in size to your own, you’ll be able to determine your SEO budget more accurately. More importantly, this research can help you identify potential opportunities—weaknesses in a competitor’s brand, for instance, or a gap your business could potentially fill. 

The Right Budget Doesn’t Break the Bank

Taking into account all of the above, we’ll wrap things up with one final piece of advice. The right budget is one that you can comfortably afford while still generating a reasonable return. Keep that in mind, and everything else should easily fall into place. 

Should You Consider Targeting Keywords With a Low Monthly Volume?

This story should be familiar to most of you.

You’ve come up with an excellent idea for content—something you’re confident will resonate with your audience and bring in qualified leads. However, when it comes time to start researching keyword permutations for that topic, you’re met with a rather unpleasant surprise. No matter how you phrase or rephrase your terms, no one seems to be searching for the topic. 

Back to the drawing board, right? 

Maybe not. As it turns out, low-volume keywords can be just as valuable as low-hanging fruit, if not more so. As noted by marketing expert Neil Patel, this comes down to a few factors:

  • Clear intent. This means that any content to which the search phrase is connected will be highly relevant. 
  • Low difficulty score. As you may already know, the lower a keyword’s difficulty score, the easier it is to rank for that keyword. 
  • Length. A long-tail keyword that contains a secondary, more valuable or higher-volume keyword may not appear worth targeting on the surface. However, it can still generate considerable traffic—and again, there’s the matter of intent. 
  • A highly niche topic. Some topics are going to be low-traffic no matter what you do—for example, a blog that exclusively targets business brokers and their clients may appear to primarily contain low-volume keywords. But the people who search for those keywords have high intent. 
  • Low cost per click. A keyword’s cost per click is another indicator of its competitiveness. Lower means the keyword has less competition, meaning it’s easier to rank for it. 

There’s also one more factor to consider—your competition. Competitors are very likely taking the same approach as everyone else, targeting relevant keywords with a reasonable search volume and difficulty score. 

By switching your focus to lower-volume keywords, you might well be able to get the jump on them from a marketing perspective. As long as you understand your audience, what they want, and what they search for, low-volume and non-competitive keywords can be just as valuable as high-traffic keywords. In some cases, more so. 

“Ignore the metrics that everyone else is using to select their keywords,” advises SEO expert Dmitri Dragilev. “Instead, focus on keywords where you can intercept the customer in the middle of the purchase decision, piggyback on the authority of an established player in a related field, [or] offer a better solution to a problem posed by an established player in your field.” 

At the end of the day, targeting low-volume keywords is really a matter of quality over quantity. After all, which would you rather have for your business? 

  • Content that brings in 50 qualified leads but generates no additional traffic. 
  • Content that brings in thousands of leads, but none of them convert. 

It’s simple mathematics at that point, really. 

Search Engine Optimization Is Not A Magic Bullet. You Need to Understand Its Limitations

We’ve all met at least one search engine optimization (SEO) snake oil salesman. You know the type. 

Grandiose and sweeping promises. Language bogged down with so much jargon it’s functionally meaningless. An endless barrage of gaslighting and cold opens. 

To hear these people talk, SEO is some sort of mystic art, and mastery means you’re guaranteed to dominate the search engine results page (SERP). 

Anyone who’s spent even a little time studying the craft knows this to be a blatant lie. SEO is valuable, indeed—it’s a powerful lead generation and marketing tool in the right hands. But it’s not some secret weapon, and it won’t allow you to seize control of Google’s algorithms. 

In order to leverage it effectively, you need to accept that—you need to understand the limitations of SEO.

It Can’t Save Low-Quality Content

All of Google’s most recent algorithm updates have been deployed with the goal of making the search engine better at recognizing whether content is valuable to the audience. Rather than operating exclusively on keyword matching, the search engine is increasingly focused on intent. It’s focused on understanding what the searcher wants and providing them with the content that best fulfills what they’re looking for. 

For this reason, if your content is poor quality, it doesn’t matter how much time you put into SEO. It’s not going to generate any meaningful returns. 

Google’s Algorithms Are Mercurial, at Best

Google releasing an algorithm that completely upsets our understanding of SEO and penalizes countless websites is very nearly an annual tradition at this point. It’s easy to forget that, regardless of how much effort we put into optimization, we’re ultimately at Google’s mercy. A single algorithm change could wipe out our progress. 

It Doesn’t Provide Immediate Returns

Unlike other paid promotion strategies, SEO is more of a slow burn. It rarely generates fast traffic or an instant return on investment. Instead, it’s more about gradually cultivating your website, building up a reputation and a rapport with high-quality content, and ensuring that content is seen by targeting the right keywords. 

It takes patience, in other words—and some people lack that patience. 

It Might Not Be A Secret Weapon, But SEO Is Still Valuable

We’d like to conclude with a bit of a disclaimer. We spent a lot of time today talking about the limitations and drawbacks of SEO. We are by no means trying to say that SEO isn’t worthwhile. 

Quite the opposite, in fact.

Even though it’s relatively slow and heavily relies on Google’s algorithms and inbound marketing content, an effective SEO strategy is ultimately a cornerstone of every successful business. It’s not a magic bullet or some holy grail of marketing. But it’s still more than worth exploring. 

What Role Does Page Speed Play in Search Engine Optimization?

Faster is better. 

That’s been common knowledge for a while now. Unless there’s a noticeable gap in quality,  a faster website will almost always outperform slower sites in the same niche. We aren’t solely talking about search engine optimization (SEO), either. 

More than half of mobile users abandon a website if it takes longer than three seconds to load. Bounce rates increase exponentially for each additional second it takes a page to load, topping out at 123% for 10 seconds. On top of this, for every second above the average a page takes to load, customer satisfaction plummets further. 

All this is to say that if your website is slow, you’re getting hit from multiple angles—by Google’s algorithms as well as a dissatisfied audience. 

How to Optimize Your Website for Speed

So, we’ve established the importance of page speed. Let’s wrap things up with a brief overview of what you can do to optimize yours. We advise the following while also using Google’s PageSpeed Insights to check on your progress as you do: 

  • Minimize HTTP requests wherever possible. 
  • Implement a content delivery network positioned geographically close to your primary audience. 
  • Leverage browser caching. 
  • Use adaptive web design, including both images and layout. 
  • Compress your website’s content. 
  • Minify your code, and only use scripts where absolutely necessary.
  • Avoid using rich media where possible.
  • If you must use JavaScript or rich media, asynchronous loading is your friend.  
  • As an addendum to the above, defer resource-heavy files so that they only load after the most crucial elements on your site. 
  • Work with your host to improve your backend and minimize time to first byte.  Note that this may require you to choose a new web host if your current one does not meet your needs. 
  • Consider running a compression audit with a tool like GIDNetwork
  • If you’re using a content management system such as WordPress, assess your plugins and uninstall any that you don’t absolutely need. Factors to look for include: 
    • Overlapping functionality.
    • Tasks that can be easily accomplished manually.
    • Poor performance optimization. 
    • Lack of consistent updates.
    • Security flaws.
  • Cut down on redirects. 
  • Optimize for mobile first, desktop second. 

We realize that’s a bit of a comprehensive list. But we also promise it’s worth the effort to incorporate every piece of advice. Not only will Google’s algorithms like your website a lot more, your audience will thank you for your efforts. If you want to see other ways you can increase the SEO performance of your site, check out our post on the role of Backlinks in SEO.

Remember that you also don’t need to incorporate all of our advice overnight. SEO has always been a gradual, ongoing process. This is really no different—ironically, if you try to rush your page speed optimization, you might end up shooting yourself in the foot before you even manage to get off the ground. 

Struggling with a Search Engine Optimization Skill Gap? Here’s What You Can Do

Given that it’s been plaguing the IT sector for years, it was inevitable that the ongoing digital talent shortage would eventually sink its talents into marketing and search engine optimization (SEO). As noted by MarketingWeek, upskilling has become crucial for specialized agencies and ordinary businesses alike. An annual survey from online marketing agency Verbli, meanwhile, found that SEO expertise is one of the most desirable skills in marketing.  

It follows that, if you currently lack that expertise internally, you need to start looking for ways to bridge that gap. We’ll walk you through how to accomplish that.

Start With a Decent Tool

Tools like Moz, Ahrefs, and Semrush can seem incredibly overwhelming at first glance.  But they also provide an excellent—albeit temporary—solution to a lack of internal SEO knowledge. All three tools are relatively simple to use once you get past the initial growing pains, and all three organizations maintain comprehensive knowledge bases to help you and your employees learn the basics of SEO.

In some cases, this might even be enough. These tools could serve as the foundation from which your business builds up skills in everything from content marketing to PPC advertising. On the other hand, you might find that you simply aren’t getting your money’s worth. 

At this point, you have three options—though there’s no rule saying you can’t attempt all three:

  • Initiate a hiring push to bring in the necessary expertise
  • Hire an agency to manage marketing and SEO for your business
  • Implement training and reskilling programs to develop SEO professionals internally 

Handling Things Yourself

If you’ve decided to take the talent acquisition route, start by looking at a few job listings for similar roles in your industry. This will give you an idea of what constitutes a competitive offer, as well as what to look for in a candidate. Ideally, you’ll want to hire someone with a proven track record—but failing that, they should at least be able to demonstrate a baseline knowledge of SEO. 

If instead you choose to develop SEO skills within your existing staff, start by looking at materials like Moz’s Beginner’s Guide to SEO. Consider how you might translate that into hands-on, self-directed training courses—if possible, allow anyone interested in this training to engage with it while on the clock. 

Bringing In an Agency

group of people sitting in front of computers

Believe it or not, hiring an agency has a great deal in common with hiring someone internally. Here as there, you’re looking for someone with a proven track record, an established client base, and clear expertise. A prospective agency should be willing to sit down with you in a discovery call to hash out your requirements, expectations, and budget. 

Be wary of anyone who over-promises. We’d also advise establishing at least a baseline understanding of SEO so you’re able to spot the scammers. Their knowledge is often only surface-level, and they rely on their targets knowing even less than they do. Want to learn more about how to avoid missing out on opportunities to increase traffic with SEO? Check out our blog where we cover SEO-related topics monthly!