What is robots.txt.?
The robots.txt is a Standard which is used by websites to communicate with web crawlers and other web robots. Which is also known as robot exclusion standard or robot exclusion protocol.
Which helps to specifies how to inform web robot about which areas of the website should not be scanned.
In simple way, robots.txt is a standard code which gives information to the web robot (search engines) to which area of website is should scan and which souldn’t.
With help robots.txt file you can hide website areas (such as wp-admin) from search engines . So search engine will not index your hide areas of your website.
If you are adding robots.txt to your website then read carefully. Because one small mistake in site can loose your website traffic.
This brief introduction about robots.txt. for more information click here Robot.txt
In this article we will know how to add robots.txt to the website.
Follow the instructions:
First login to your wordpress admin dashboard.
Here I am using Yoast SEO plugin. If your are using different one then finding Tools section might be different. But rest of procedure is same.
Also Read : Yoast SEO (must needed plugin).
Now click on SEO section which is on sidebar. Now go to TOOLS >> File Editor.
Click on File Editor.
After clicking File Editor section you will find Create robots.txt file. Just click on that.
Which will create a robots.txt file.
Just copy the below robots.txt file and paste it below the generated robot.txt.
( Sitemap: https://example.com/sitemap_index.xml )
Change the example.com by your Domain name.Now Click on Save changes to robots.txt.
it completes all your wordpress works.
Now its time to add robots.txt on Google Webmaster tool /Google Search Console.
For this login into search console. Now you have to go to Crawl >> robots.txt tester in old version of tool.
The code here is different from our website code. So simply click on See Live Robots.
Which wil show your updated robots.txt. just copy and paste it.
After pasting the robots.txt file click on Submit.
Now you have to choose Ask Google to update. And submit.
Here your work is done.
To know whether Google is indexing is or not your post. Just copy one of your post URL and paste it in box.
Click on TEST. If it is shows ALLOWED then Google is indexing your post.
You can also check for your wp-admin. Which will definitely shows BLOCKED. Because in robot.txt we Disallow the wp-admin. That means we are hiding wp-admin from Google indexing.
That’s it, I Hope!! This article will help you to know How add robots.txt file to website.
If you like this article then share your Comments and Subscribe to get new stuffs like this.