Guide to creating and setting up the right Robots.txt for the site. Description of setting Allow and Disallow site indexing directives in Robots.txt. Robots.txt file optimization examples for CMS Wordpress.