- February 8, 2017
- Posted by: user
- Category: Uncategorized
If you want to hide a page or website from search engines, you can do it in several ways. Since all search engines follow a web robots standard while crawling websites called Robots Exclusion Protocol, you can use the robots.txt file to give instructions to a search engine on what to index and what not to index.
First, create a text file called robots.txt on the root of your website. Now let’s say you want to block your entire website from being indexed by the search engines, so you would add these lines to your text file:
[code:xml]User-agent: *
Disallow: /[/code]
You can also block directories or individual pages on your site using a robots.txt file instead of blocking the entire website. To block a directory, you could add the following lines:
[code:xml]Disallow: /cgi-bin/
Disallow: /tmp/[/code]
You can also block access to your website or webpage using META tags. To prevent all robots from indexing a page on your site, add this line to the HEAD section:
[code:xml]<meta name="robots" content="noindex">[/code]