How Is the Robots.txt File Used for SEO?

The robots.txt file is an important part of search engine optimization (SEO) that can help improve your web pages’ visibility in search engines. This article will describe what the robots.txt file is, how it works and how you can use it to improve your site’s SEO. We’ll also look at some best practices and tips for making the most of this powerful tool.

What Is Robots.txt?

Robots.txt is a text file that tells bots, including search engine crawlers, which pages and files on a website they should be able to see and index. Its main job in SEO is to instruct search engine crawlers like Googlebot not to look at certain pages. 

For example, you might want to tell them not to look at the category pages of your blog or online store because you don’t want those pages in search results. Other uses include preventing the indexing of duplicate content and other files you may not want indexed, including scripts and images.

Robots.txt can help ensure that your site’s crawl budget is used efficiently. Search engines tend to limit the number of pages they crawl each day. Site owners often want to focus their crawl budget on specific pages, and the robots.txt file can be used to ask Google not to use it on less important pages.

Does Robots.txt Stop Bots from Indexing Pages?

Well-behaved bots follow the directives in your robots.txt file. However, not all bots are well-behaved, and those that aren’t crawl whatever they like. Fortunately, Google and other major search engines usually obey robots.txt instructions.

But even if you use your robots.txt file to stop crawling, Google will still be able to see the pages via external links, so they may end up in search engine results anyway. If you want pages to be on the web but invisible, Google suggests password-protecting them. 

You can also use the noindex tag to prevent indexing. But don’t use robots.txt and noindex together to block pages—if Google can’t crawl a page because it is blocked in robots.txt, it can’t see the noindex directive and therefore may index the page and show it in search results. 

Useful Robots.txt Directives for SEO

Here are four robots.txt directives you may find useful for search engine optimization:

  1. User-agent: * – Directives are divided into blocks, each of which begins with a user-agent line indicating which user-agent (crawlers, browsers, and so on) it applies to. The asterisk (*) indicates all user-agents, but you can name specific agents like “Googlebot.”
  2. Disallow: /cgi-bin/ – This directive prevents search engine bots from crawling and indexing the files in the cgi-bin folder. 
  3. Allow: /images/ – This directive tells search engine bots that they are allowed to crawl and index the files within the images folder. Allow directives are used to override disallow directives. For example, you may want to allow access to a folder inside a disallowed parent folder. 
  4. Sitemap: – This directive specifies which sitemap should be used, helping bots discover new content faster and more accurately.

In a robots.txt file, these directives would look like the following:

User-agent: *

Disallow: /cgi-bin/

Allow: /images/


Make Sure Your Robots.txt File Is Error-free

A word of warning: Make sure your robots.txt directives precisely follow the specification. Mistakes result in unexpected behavior that could hurt your site’s SEO. To be sure, run your robots.txt file through a validator.

Crucial Last-Minute SEO Practices for Boxing Week

Boxing week is just about here. You’ve sprinkled your store with decorations, you’ve decided which products you want to put on sale, and you’ve done the necessary legwork to spread the word about your upcoming promotions via email, paid advertising, and social media. But is your website ready for the influx of traffic it’s likely to receive?

Don’t panic if the answer is no. If you’ve laid the groundwork with promotions and sales, you’ve already done the heavy lifting. All that’s left is the digital equivalent of making sure a brick-and-mortar store is clean and accessible the night before a major event. 

At the end of the day, that’s really all search engine optimization (SEO) involves. With that in mind, we’ve put together a checklist of last-minute SEO practices and tactics to prepare your store for its boxing week sale. 


  • File names and alt text both use keywords describing the product as plainly as possible. 
  • All image files are properly sized (per Shopify).
    • A maximum of 1 megabyte (MB) for header and background images.
    • A maximum of 300 kilobytes (KB) for product photos.
    • 100 KB or less for product thumbnails.
    • A maximum of 10 MB for hero images/feature photos. 
  • All image files are uploaded in the proper aspect ratio.
    • 16:9 for hero images.
    • 1:1 for product photos and thumbnails.
  • Files are uploaded as jpegs. 
  • All product photos are of reasonable quality and give an idea of how each product looks. 

Product Pages

  • Product titles include all relevant information, including manufacturer name, model/color, and SKU.
  • Product descriptions provide a complete overview of what each product is and how it works. 
  • Each product page has a unique description.
  • Product page URLs are simple and easy to remember. 
  • All product pages use the Product schema. 
  • Each product page is tagged with all relevant keywords for internal searches. 
  • User reviews are enabled. 
  • You’ve used a keyword research tool to determine the best keywords for each product. 

General Performance and Usability

  • You’ve tested your website with Google PageSpeed Insights and made any recommended adjustments.
  • You have asked your web host if it’s possible to scale up your bandwidth for the week. 
  • All content is optimized for mobile devices. 
  • Site navigation is streamlined and intuitive, allowing users to find what they’re looking for in as few clicks as possible. 
  • There are minimal interruptions to the user experience—no pop-ups or pop-over prompts. 
  • You offer multiple payment and shipping options at checkout.
  • Your website features an SSL certificate during checkout. 


Too many e-commerce website owners kill it when it comes to off-site promotion while neglecting the basics of on-site optimization. With the checklist above, you can ensure that you aren’t one of them—for boxing week as well as every event that follows. 

Go Beyond Traditional User Intent With Micro-Intents

We’re going to assume you’re already familiar with the four major search intents—transactional, informational, navigational, and brand.  What you may not know is that as search engine optimization has continued to evolve, those intents have slowly started to lose relevance. To put it another way, a general idea of what a searcher wants is no longer sufficient. 

You need specifics. What precise content does someone with informational search intent want to find? At what stage of the buyer’s journey is a user with transactional/commercial intent? 

That’s where micro-intents come in. As noted by Search Engine Land, these subcategories of traditional search and user intents offer a better understanding of audience expectations. And that, in turn, can be used to create more targeted, higher-value content.

We’ve listed the different micro-intents below for posterity.  

Informational Micro-Intents

  • Entertainment. Entertainment-focused content is all about passing the time. It can take many different forms but is typically easily digestible. 
  • Definition. If you want an example of definition-focused content, you’re reading it. You came here because you wanted to know what micro-intents are. 
  • Expansional. This type of content is similar to definition-focused but takes a much deeper, more comprehensive dive. 
  • Enablement. Typically, enablement content takes the form of how-to articles and videos. 
  • Overview/Aggregation. Similar to definition/expansional, overview content takes a high-level look at a topic. Examples include infographics and listicles.  

Transactional/Commercial Micro-Intents

  • Comparison. The user is looking to buy but wants to determine which product/brand is the best before they do so. 
  • Category. The user has a general idea of the product or service they want but is still trying to decide on a specific solution. 
  • Product. The user is at the bottom of the funnel and is about to make a purchase—they’re looking to find out a bit more about a product or service before they finalize their decision.  

Brand Micro-Intents

There are actually no specific brand micro-intents to speak of. Generally, branded searches can be positioned under one of the other three categories. With that said, branded searches are typically looking for reviews, testimonials, or other information—anything that one would typically use to build trust. 

Navigational Micro-Intents

  • Support. The user is currently a customer and is looking for help with their product or service. This could take the form of instructional articles, a product-related knowledge base, or contact information for a support professional.
  • Website. Self-explanatory. The user wants to find a specific page, blog post, or social channel. 
  • Location. Similar to the previous intent, except that they’re looking for a location in the real world rather than online. 

How Do You Determine Micro-Intents? 

The good news is that micro-intents are no more challenging to determine than high-level intents. Simply pay attention to specific phrases or keywords, and use your best judgment. From there, it’s simply a matter of creating content that’s more directly targeted. Couple this with the information we provided on identifying keyword targets, and you’re sure to find success in your SEO efforts.

Three Things You Need to Understand About TikTok SEO

With over one billion monthly active users, TikTok is among the most popular social networks in the world, eclipsed only by Facebook, YouTube, and Instagram. It’s also rapidly becoming the social app of choice for millennials and generation Z, who together make up roughly 75% of its global base. Suffice it to say, if your brand is looking to target either of those audiences, it’s in your best interest to gain a foothold. 

That’s actually not as difficult to do as you might expect—or at least, it’s no more or less challenging than any other social network. As is usually the case, success starts with a solid marketing strategy and a decent understanding of search engine optimization.  With that in mind, here are the three most important things you must understand about TikTok SEO.  

The Basics of Topic Ideation Remain the Same

Plenty of brands treat social media as some sort of arcane, unknowable entity, as if they’re convinced they’ll need to reinvent the wheel in order to succeed. The reality, however, is that the foundation of TikTok SEO, topic ideation, is the same as on any other site. To summarize: 

  • Understand your audience—who they are, what they’re looking for, and why they might be interested in your brand. 
  • Leverage sites like Quora and tools such as Answer the Public to research the questions your audience is asking and the topics that interest them.  
  • Pay attention to how any topics you identify relate to your brand’s niche. 
  • Identify the specific keywords in those questions and topics, then leverage them in your TikTok videos, captions especially. 
  • Be respectful of your audience’s time. There’s rarely a reason for a TikTok video to be longer than a minute. 

Trends and Hashtags Are Crucial for Discoverability

Especially with Google’s eventual plan to index TikTok video content, choosing the right hashtags and trends goes a long way toward increasing your brand’s reach and visibility. Successfully leveraging more popular trends can generate a torrent of views and engagement, potentially even causing your page to go viral. Provided you’ve targeted the right users and linked your TikTok channel back to your website, that also has the potential to lead to a high volume of conversions. 

Quality and Consistency Always Win

The phrase ‘content is king’ may be overused, but that doesn’t make it any less true. High-quality, relevant content is the most important factor in your success on TikTok—or really, on any channel. That means crystal-clear audio and video and a well-written script geared towards your audience. 

Without those characteristics, it doesn’t really matter how well you optimize your TikTok videos from a search perspective. Reach is all well and good, but only if it’s attached to content that people actually want to watch. Always keep that in mind.

And above all, remember to always create content with purpose. 

Horizontal Content vs. Vertical Content

If there’s one thing about any industry that always holds true, it’s that people love their buzzwords. And while such industry parlance can make things easier for someone who’s been around for a while, it can be downright overwhelming for newcomers to the field. That’s why we intend to keep doing our part to clear the air—to walk people through all the definitions bandied about by marketing professionals.

To that end, today we’re going to discuss the difference between horizontal content and vertical content. As you’ve probably guessed, these two phrases have a lot in common with business markets, at least in that they define scope in a very similar manner. 

man writing on macbook

What Is Vertical Content? 

The clue is in the name—vertical content is, quite fittingly, a catch-all term for any piece of content targeted at a specific industry, market, or niche. Typically, when someone discusses vertical content, they’re specifically focused on business to business marketing. However, this need not always be the case.

A vertical is any specialized niche with a highly-defined audience. A healthcare cybersecurity publication and a Portland-based craft brewing business will both publish vertical content as part of their inbound marketing efforts. The thing that defines vertical content is specificity.

Its audience is incredibly well-defined, and it’s incredibly unlikely that anyone outside of that demographic will have any interest in a vertical-focused business or its content. 

What Is Horizontal Content?

While vertical content is highly specific, horizontal content is a little more generalized. Rather than targeting an exclusive market or sector, horizontal content focuses on broader problems or needs. Again, although horizontal content is frequently targeted at the consumer sector, there are plenty of business audiences for which this content is beneficial, as well.

With that said, since horizontal content is by its nature a bit more nebulous than vertical content, we’ll offer up a few examples of topics that might typically have a horizontal focus:

  • General cybersecurity.
  • Consumer electronics.
  • Grocery. 
  • Renovations.
  • Home decor. 
  • Health and wellness. 
  • Productivity. 

Choose the Best Content for Your Needs

It’s important to understand that neither of these content types is superior to the other. Some businesses are far better suited for horizontal content, while others, by their nature, will see the most success with vertical content. It’s also worth noting that in most cases, you can take a horizontal topic and narrow it down to a particular vertical and vice-versa. 

The best advice we can give you here is to first define your customer base and their needs. Once you understand who you’re marketing to and what their interests are, it will be a far simpler matter to figure out what time of content to publish. Just remember that at the end of the day, what truly matters is that you provide value to your audience—everything else is secondary to that. 

Wondering Why You’re Not Ranking on Google? Here’s Three Possibilities

If your content isn’t ranking—or if it was ranking but has suddenly dropped down the SERPs—you’re generally left up to your own devices when it comes to figuring out what went wrong. In our experience, however, the answer usually isn’t all that complicated. When content isn’t ranking on Google, it usually comes down to one of three things. 

You’re Targeting the Wrong Keywords

Each and every piece of content on your site should be built out of the following questions

  • Why am I creating it? 
  • What value does it provide to my audience? 
  • What search terms would someone typically use to find this content? 

The last question is where we generally see people slip up. They might be targeting keywords that are too competitive, or trying to rank for phrases dominated by a much larger, more successful brand. Or maybe they’ve chosen keywords based on raw numbers rather than intent

Either way, if your content isn’t ranking, your first step should be to re-examine your keywords. 

Your Website is Bad

Picture two department stores.

  • The first store is clean, well-lit, and laid out in a streamlined, intuitive way. The way everything is laid out makes sense to shoppers, and the staff are friendly and efficient.
  • The second store is a dirty, confusing labyrinth. No one is entirely sure where anything is, and employees are surly at best, actively unhelpful at worst. 

Where would you rather shop? 

The best websites load fast and have an interface that’s easy to both navigate and understand. Their content is well-organized, and their design is aesthetically pleasing. Can you say that about your site, too? 

You Simply Aren’t Publishing Compelling Content

Not everyone is a content creator—and not everyone needs to be. If you know for a fact your keyword research is accurate and your website is well-designed, it may well be that your content simply isn’t that good. Remember that the best content is:

  • Original
  • Concise
  • Informative and/or entertaining
  • Well-written, with no errors
  • Trustworthy
  • Authentic

If you’re simply rehashing word for word the things you’ve found elsewhere, you’re not going to rank. 

Closing Thoughts

If there’s one thing that’s remained constant about search engine optimization for the past few years, it’s Google’s insistence on keeping its search algorithms close to its chest. Much of what we know about search engine optimization (SEO) and the search engine results page (SERP) is the result of observation and educated guesses. It’s conjecture made by knowledgeable experts, but conjecture all the same.

That isn’t to say that Google has provided no SEO guidance, mind you. The company has actually published a fairly comprehensive SEO Starter Guide that walks you through pretty much all the basics. The last piece of advice we’ll give is to familiarize yourself with that guide if you’ve not done so already.

It may help immensely.

Understanding the Different Types of Keywords

Keywords used to be simple. You’d enter your search terms, and you’d be presented with a list of results containing your keyword or phrase. Pages with more instances of that phrase ranked higher on this list. 

As you might expect, this system was incredibly easy to abuse. That’s why since then, both keywords and the algorithms that analyze them have grown considerably more complex. These days, there are a ton of different classes of keywords—and it’s in your best interest to understand them all. 


Let’s start with the different matching options you can choose during keyword research and ad targeting. Once you’ve defined your initial topic, focus, or core keyword, you can find variants of it in a few different ways. 

Broad Match

Although search engine optimization research tools still allow broad match searches, broad match keywords have been retired by Google as of July 2021. Originally, they used to match any result containing your core keywords. They accounted for any permutation or variation of those words, and also included similar words. 

This functionality has largely been wrapped into phrase match keywords. 

Phrase Match

Phrase match keyword searches contain your core keywords in the order you typed them. When it retired broad match keywords last year, Google also tweaked phrase match keywords. Phrase match searches now apply road match criteria while still preserving the original meaning of the search. 

Exact Match

An exact match keyword is exactly what it sounds like. The exact permutation of each word in the exact order you typed them. With that said, exact match targeting will also typically include reordered or rephrased close variants if Google determines that they retain the original meaning. 


Arguably the most important characteristic of a keyword is its intent—what the searcher is looking to do.


Someone searching with commercial intent fully intends to make a purchase as soon as possible. They’re effectively right at the end of the sales funnel, ready to convert when they find what they’re looking for. 


Transactional intent is similar to commercial. Searchers do intend to make a purchase at some point in the future, but they’re still doing research. 


Someone searching with informational intent simply wants to learn. They might be looking for a how-to, developing their own content, or performing academic research. Though they may be convinced to convert, they typically aren’t looking to purchase. 


A person searching with navigational intent is looking for a specific website or brand. They know exactly what they’re looking for, and why. Your job is to ensure your site provides this to them. 


Lastly, keywords can be categorized in terms of length. 


A short-tail keyword tends to be much more general and consists of only one or two words. Generally speaking, a short-tail keyword acts as a starting point. Users don’t typically only type one or two words into Google, instead resorting to more generally conversational searches. 


Long-tail keywords consist of three or more words. Particularly with the growing prominence of semantic search, long-tail is generally the way to go. 

Link Building 101: Laying the Foundations for Outreach

It’s a bit of an oversimplified explanation, but the more high-quality pages that link to one of your pages, the greater that page’s authority. The greater that page’s authority, the better it’s likely to rank on Google. The higher your PageRank, the more traffic you get through organic search.

When it comes to building up your website and creating brand awareness, link building is arguably one of the best strategies at your disposal—but you must first understand how it works. 

What is Link Building? 

Link building is the process of generating more inbound links to your website. Said links must be from websites that themselves have a reasonably high E-A-T score—Expertise, Authoritativeness, and Trustworthiness. In simple terms, what this means is that they have an established reputation for publishing authentic, informative, high-quality content. 

How Do You Start Link Building?

As with any marketing or growth strategy, the first step in any link building campaign is to define what you want to accomplish with it. 

Are you trying to establish yourself as an authority in your sector? Are you looking to generate conversions and inspire more people to purchase your products or services? Or do you simply want to generate traffic and increase awareness of your brand? 

What Are Some Common Link Building Strategies? 

Modern link building is generally predicated on a very simple idea—if you create excellent content, people will link to that content. Informative, entertaining, and well-researched blog posts can easily generate a ton of passive inbound links to your website, particularly if they rank well on the search engine results page. Similarly, highly shareable content can draw considerable traffic from social media. 

However, if you want to take a more active role in your link building efforts, you generally have three options. 

  • Guest blogging. Find a well-established site in either your niche or a related one, and create guest content that can be published on that site with a link redirecting back to you. Typically, it’s advisable to focus on sites with guest editorial programs, but that doesn’t mean you can’t reach out if you find a promising prospect—but do not, under any circumstances, use a generic email template. 
  • Seek out broken links. Basically, this involves looking at high-quality sites and searching for broken or outdated links. When you find one, contact the webmaster and offer to replace the link with similar content on your own site. 
  • Syndicated content. Somewhat similar to guest blogging, this involves creating high-quality content such as an infographic, then submitting it to user-driven directories and content sites. 

What Link Building Tactics Should I Avoid? 

Let’s wrap things up with a quick list of link building tactics that are not only doomed to fail, but likely to get your site penalized in the process: 

  • Buying your inbound links.
  • Using a link farm or content mill.
  • Paid content disguised as organic content.
  • Purchasing/creating and interlinking multiple websites.
  • Spamming links in comments, forums, etc. 

Protecting Against SEO Poisoning

In recent months, a tactic from the earliest days of SEO has re-emerged. Here’s how you can deal with it, as both a website owner and an Internet user.

As the old saying goes, if it ain’t broke, don’t try to fix it. 

Unfortunately, that applies to the tactics used by cybercriminals just as much as it applies to legitimate businesses. It’s why distributed denial of service (DDOS) attacks have been around for decades. And it’s why ransomware has changed so little over the years.

Sure, we’re seeing more sophisticated distribution tactics, but ultimately modern ransomware works nearly the same as ransomware from a decade ago.

With that in mind, it appears another blast from the past has started to re-emerge. At least a few of you probably remember the early days of search engine optimization (SEO). Back when search engines were akin to a virtual wild west, black hat tactics were the best way to rank. 

What Is SEO Poisoning? 

You likely also recall how many malicious websites rose to the top of the search engine results page (SERP), abusing SEO to serve poisoned results to users, hence the term SEO poisoning. It didn’t take long for Google and other search engines to release algorithm updates that shut down most of the shadier tactics. But just like life, criminals always find a way. 

Today, SEO poisoning takes a different form.

First, criminals create a website and take great pains to make it look as legitimate as possible. From there, they begin ‘trend chasing,’ leveraging their understanding of SEO in an effort to gain a prominent position on the SERP. There’s no limit to the number of keywords a single malicious domain may target in this manner.

Security firm Websense Security Labs estimates that these malicious websites represent as much as a quarter of the first page of search results for trending topics.

Generally, the objective is fraud or identity theft. Hackers will use the poisoned sites to steal the personal details of unsuspecting users. They might also inject ransomware onto a victim’s system, add another node to a botnet, or — if they’re lucky enough to infect a PC belonging to a webmaster — hijack another website to add to their malicious network. 

How Do I Protect Myself From SEO Poisoning? 

As with many types of cyberattacks, a little mindfulness goes a long way.

  • Be incredibly wary of opening a website you’ve never heard of before, particularly if you’re searching for a trending topic.
  • We also strongly recommend installing an up-to-date antivirus.
  • Use a password manager for both your personal accounts and your business accounts.
  • Keep all your software and systems up to date. 
  • It may be worthwhile to use a VPN or invest in a router that has built-in encryption. 
  • Consider installing an ad blocker and blocking scripts, as ad networks and malicious scripts are two prevalent delivery mechanisms.

If you own or operate a website, the same rules apply — remain vigilant, and put in the necessary work to keep your personal files and your professional data safe. 

Measuring the Return on Your Search Engine Optimization Efforts

Search engine optimization is a must for any business with a web presence. But how do you determine whether or not your efforts are bearing fruit?

As with any business initiative,  it’s crucial that you understand how to quantify the success of your search engine optimization efforts. Not only do you need to show leadership that your budget is generating tangible results, but you also need to calculate SEO’s return on investment for your own purposes. Consistent measurement of ROI can also help you identify weaknesses and shortcomings, and help you determine where your time (and money) should be directed. 

But how exactly do you measure the ROI of SEO? 

Nebulous Returns

From an ROI perspective, the issues with SEO are similar to those with marketing. Namely, while there are certain fixed costs and returns, as a whole, the core goals of SEO are difficult to express in concrete numbers. Concepts like brand awareness and organic visibility are inherently abstract.

With that said, it is possible to at least approximate them. 

SEO Metrics That Measure ROI

The first step in determining ROI is to figure out how much you’re spending on SEO. If you’ve hired a third-party agency, this is relatively easy to calculate. Just look at whatever you’re paying them. 

If you’re managing SEO internally, things get a bit more complicated, and you’ll need to look at a few different factors. 

  • How much time your staff is spending on SEO. This includes developers, designers, marketing specialists, etc. 
  • How much per hour each staff member is paid, on average. 
  • Subscription costs for any tools or platforms you’re using to inform your SEO efforts. 

With those numbers in mind, measure the following key performance indicators (KPIs) from the beginning of your SEO campaign to its end: 

  • Organic traffic. Self-explanatory. Traffic generated from the search engine results page (SERP).
  • Bounce rate. The number of people who visit your site and leave without performing any actions. Can be paired with time on site to identify potential bottlenecks. 
  • Organic impressions. How many people have seen your site on the SERPs. 
  • Organic click-through rate. The number of users who clicked your site on the SERP, measured against total impressions. 
  • Pages per session. How many pages a user views, on average. 
  • Conversions. Here’s where things get a bit complicated, as there are multiple ways you might define conversions. 
    • Sales. 
    • New subscribers. 
    • Sign-ups for mailing lists/asset downloads. 
    • Downloads. 
    • Social shares. 
    • Phone calls. 
    • Demo/proof of concept requests. 

One way to monitor the above is through conversion tracking. Google allows you to define certain actions as conversions, while also assigning a dollar value to each. Although this is typically intended for Google Ads, it can easily be applied to your site. 

Expressing the ROI of SEO

So, in light of the above, you can express the value of your SEO efforts in a few different ways: 

  • Percent increase/decrease. Applies to bounce rate, organic traffic, impressions, clickthrough, etc. 
  • Spend vs. Revenue. Specifically applies to conversions that involve monetary transactions. 
  • Volume. How much more traffic/how many more phone calls you’re receiving now versus when you started.