Add Sitempal information in to Robots.txt File

There’s nothing new about a robots.txt file, but I have started using robots.txt validators that will not validate unless an XML sitemap is specified.
This is very good practice as the XML sitemap will get picked up by all crawlers that support the tag ‘Sitemap’ within your robots.txt file (Google and MSNBot do for starters).
So using as a reference my example robots.txt file will look like this:



User-agent: *

Allow: /wp-admin/admin-ajax.php

Disallow: /wp-admin/

Disallow: /directory-x

Disallow: /directory-y

The blank link between the Sitemap tag and the User-agent tag is essential as this can cause problems for some
crawlers. However this method will gurantee your website indexing with major search engine.
Happy SEO!