Setting up the correct settings for crawlers and indexing in Blogger is crucial for search engine optimization (SEO). Hence, the best settings will enable your website to rank in a short time.
There are few Plugins available for SEO in Blogger, so you have to use every option available to index and rank your website on search engines like Google and Bing faster.
Crawlers and indexing settings on Blogger can play a significant role for your Blogger website to get indexed fast, so you have to set it up properly.
What is a Crawler?
A crawler, also known as a spider or bot, is run by search engines like Google and Bing. It discovers and scans webpages and then indexes them in the database.
Hence, those websites can be shown in the search results.
The main objective of these crawlers is to learn and categorize the topics and contents on the webpage so that they can be retrieved whenever someone searches for them.
Suggested: How to Fix Blogger Https Availability Status Unknown or Is Being Processed?
What is Indexing?
Indexing or Search indexing is like adding books to a massive library with labels. So that search engines can easily retrieve information from it whenever someone searches on the internet.
So, your website must get indexed. Otherwise, it won't be shown in the search results.
How to Set Up Crawlers and Indexing Setting in Blogger For SEO?
In Blogger, Crawlers and indexing status settings remain untouched by default.
It does not mean your blog will not get indexed, but it will take more time to index your blog posts. Because Crawlers will crawl everything on your website, including unnecessary pages like archived pages.
So, you have to enable the Crawlers and indexing settings and customize it. It will help the crawler understand which posts and pages to crawl, and your posts will get indexed faster.
Custom Robots.txt
Log in to the Blogger dashboard, go to the blog settings, and scroll down to the Crawlers and indexing settings. First, you need to enable the custom robots.txt option.
A robots.txt file informs the web crawler which posts or pages it can request and which pages are not required to crawl. It also helps your website to avoid too many bots requests.
Now, you have to add custom robots.txt; you have to first generate a robots.txt file. You can simply copy the example below and paste it after changing the domain name.
User-agent: *
Disallow: /search
Allow: /
Sitemap: https://www.DomainName.com/sitemap.xml
Read also: How to enable AMP in Blogger?
Custom Robots Header Tags
Once you enable the option for custom robots header tags, you will see that Home page tags, Archive and Search page tags, and Post and Page tags options are available.
Let's explain the meanings of these tags in Blogger:
- All: it means there aren't any restrictions for crawlers to crawl or index any page.
- Noindex: if this setting is selected for any page, the crawler can't crawl or index that page.
- Nofollow: Crawlers automatically crawl all the links on any webpage, but if you don't want crawlers to crawl any links on any page, you have to enable this option.
- None: it means both "noindex" and "nofollow" simultaneously.
- Nosnippet: This setting stops search engines from showing snippets of your web pages.
- Noarchive: This option will remove any cached link from the search.
- Noimageindex: Your post will be crawled and indexed in this case, but all the images will not be indexed.
- Noodp will remove titles and snippets from open directory projects like DMOZ.
- Unavailable_after: this option helps to noindex any post after a particular time.
- notranslate: it will stop crawlers from showing translation on pages.
Suggested: Killer SEO Applications for Ranking a Blogger or Blogspot Blog.
How to Enable the Correct settings for Blogger Robots Tags?
Now that you know all the details about the robot tags, you can decide which one to choose. By the way, here are the best settings for any regular blog or website. Only enable these tags category-wise.
- Custom robot tags for the home page: all, noodp
- Custom robot tags for posts and pages: all, noodp
- Custom robot tags for archive and search pages: noindex, noodp
How to Manage custom robot tags for each Blogger Post
After changing the Crawlers and indexing settings, custom robot tags will also be available in post settings at the bottom right-hand side of the post writing dashboard. So, you can easily control settings for each post.
Conclusion
Crawlers and indexing settings can play a significant role in blog posts SEO, but if you do it wrong, it can also remove your posts altogether from Search results.
See Also: How to Remove/Delete Spam Links From Blogger Comments?