robots.txt Generator
Create a robots.txt file that tells search engine crawlers which pages they can and cannot request. Add user-agent rules, allow and disallow paths, and sitemap URLs.
Rules
Use * to match all crawlers, or a specific bot name like Googlebot.
One path per line. Example: <b>/public/</b>
One path per line. Example: <b>/admin/</b>
Optional seconds between requests. Not all crawlers honor this.
Sitemaps
One absolute URL per line. Example: <b>https://example.com/sitemap.xml</b>
Preview
Plain-text robots.txt output based on your rules.
Your robots.txt will appear here as you add rules.
Code
Save this file as robots.txt in the root of your website.