How to Create a Robots.txt File

Overview

A robots.txt file tells search engines which areas of your website they can and cannot crawl. While it does not directly improve rankings, a properly configured robots.txt file helps search engines understand your website structure and avoid crawling unnecessary pages.

Most websites should have a robots.txt file, especially if you want to provide search engines with your sitemap location or limit access to certain sections of your site.

What is Robots.txt?

The robots.txt file is a plain text file located in the root directory of your website.

Example:

https://yourdomain.com/robots.txt

When a search engine visits your website, one of the first files it checks is robots.txt. The instructions inside the file help guide search engine crawlers such as Googlebot and Bingbot.

Basic Robots.txt Example

The following configuration allows search engines to crawl your entire website and provides the location of your sitemap:

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

For websites using dynamic WordPress sitemaps, you may use:

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/?sitemap=1

How to Create a Robots.txt File

Option 1: Create Manually

1. Open a text editor such as Notepad or VS Code.
2. Create a new file named:

robots.txt

3. Add your desired rules.
4. Save the file.
5. Upload it to your website’s root directory (public_html).

Your file should be accessible at:

https://yourdomain.com/robots.txt

Option 2: Create Through WordPress SEO Plugins

Many SEO plugins automatically generate a virtual robots.txt file.

Popular options include:

• Rank Math
• Yoast SEO
• All in One SEO

After creating or editing the file, visit:

https://yourdomain.com/robots.txt

to verify the changes.

Common Robots.txt Rules

Allow Everything

User-agent: *
Allow: /

This is the most common configuration for public websites.

Block a Specific Directory

User-agent: *
Disallow: /private/

Search engines will avoid crawling URLs inside the private folder.

Block a Specific File

User-agent: *
Disallow: /example-page.html

Add a Sitemap

Sitemap: https://yourdomain.com/?sitemap=1

Adding a sitemap helps search engines discover your content more efficiently.

Important Notes

Robots.txt Does Not Hide Content

Many website owners mistakenly believe robots.txt protects sensitive information.

This is not true.

Robots.txt only provides crawling instructions. Protected content should use proper authentication, permissions, or noindex directives where appropriate.

Do Not Block Important Pages

Avoid blocking:

• Homepage
• Blog posts
• Product pages
• Categories
• Essential website resources

Blocking important content can prevent search engines from properly indexing your website.

Be Careful With This Rule

User-agent: *
Disallow: /

This blocks the entire website from search engines and can remove pages from search results if left in place.

How to Test Your Robots.txt File

After creating your robots.txt file:

1. Visit:

https://yourdomain.com/robots.txt

2. Verify the file loads correctly.
3. Check for formatting errors.
4. Confirm your sitemap URL is listed correctly.

You can also review Google’s robots.txt documentation for additional guidance.

Recommended Robots.txt for Most Websites

The following configuration works well for most WordPress websites:

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/?sitemap=1

Replace the sitemap URL with the correct sitemap location for your website.

Frequently Asked Questions

Do I need a robots.txt file?

No, but having one is recommended because it helps search engines understand your website and locate your sitemap.

Can robots.txt improve SEO?

Not directly. However, it helps search engines crawl your website more efficiently and can prevent unnecessary pages from being crawled.

Where should robots.txt be located?

The file should be placed in the root directory of your website, typically inside the public_html folder.

Can robots.txt prevent pages from appearing in Google?

Not always. To prevent indexing, use appropriate noindex directives instead of relying solely on robots.txt.

Need Help?

If you need assistance creating or troubleshooting your robots.txt file, contact the VeerHost support team and we’ll be happy to help.