How to add robots.txt file to website.???

0
202
views
how to add robot.txt file to website

What is robots.txt.?

The robots.txt is a Standard which is used by websites to communicate with web crawlers and other web robots. Which is also known as robot exclusion standard or robot exclusion protocol.

Which helps to specifies how to inform web robot about which areas of the website should not be scanned.

In simple way, robots.txt is a standard code which gives information to the web robot (search engines) to which area of website is should scan and which souldn’t.

With help robots.txt file you can hide website areas (such as wp-admin) from search engines . So search engine will not index your hide areas of your website.

If you are adding robots.txt to your website then read carefully. Because one small mistake in site can loose your website traffic.

This brief introduction about robots.txt.  for more information click here Robot.txt

In this article we will know how to add robots.txt to the website.

Follow the instructions:

First login to your wordpress admin dashboard.

Here I am using Yoast SEO plugin. If your are using different one then finding Tools section might be different. But rest of procedure is same.

Also Read : Yoast SEO (must needed plugin).

Now click on SEO section which is on sidebar. Now go to TOOLS >> File Editor.

Click on File Editor.

how to add robot.txt file to website

After clicking File Editor section you will find Create robots.txt file. Just click on that.

how to add robot.txt file to website

Which will create a robots.txt file.

how to add robot.txt file to website

Just copy the below robots.txt file and paste it below the generated robot.txt.

( Sitemap: https://example.com/sitemap_index.xml )

Change the example.com by your Domain name.Now Click on Save changes to robots.txt.

how to add robot.txt file to website

it completes all your wordpress works.

Now its time to add robots.txt on Google Webmaster tool /Google Search Console.

For this login into search console. Now you have to go to Crawl >> robots.txt tester in old version of tool.

how to add robot.txt file to website

The code here is different from our website code. So simply click on See Live Robots.

how to add robot.txt file to website

Which wil show your updated robots.txt. just copy and paste it.

how to add robot.txt file to website

After pasting the robots.txt file click on Submit.

how to add robot.txt file to website

Now you have to choose Ask Google to update. And submit.

how to add robot.txt file to website

Here your work is done.

To know whether Google is indexing is or not your post. Just copy one of your post URL and paste it in box.

how to add robot.txt file to website

Click on TEST.  If it is shows ALLOWED then Google is indexing your post.

how to add robot.txt file to website

You can also check for your wp-admin. Which will definitely shows BLOCKED. Because in robot.txt we Disallow the wp-admin. That means we are hiding wp-admin from Google indexing.

That’s it, I Hope!! This article will help you to know How add robots.txt file to website.

Also Read : How to add Pop-up subscription form for website.?

If you like this article then share your Comments and Subscribe to get new stuffs like this.

LEAVE A REPLY

Please enter your comment!
Please enter your name here