SEO Webmaster Crawler Builder
Robots.txt File Generator
Construct an optimized, RFC-compliant robots.txt file to control search engine indexing, protect administrative directories, and declare XML sitemaps.
User-Agent Rule #1
User-Agent Rule #2
XML Sitemaps
robots.txt Preview
# ==================================================================== # Robots Exclusion Protocol (robots.txt) # Generated by TxtCraft Pro (https://txtcraft.site) # Date: 2026-08-28 # ==================================================================== User-agent: * Disallow: /admin/ Disallow: /api/private/ Disallow: /checkout/ Allow: /public/ Allow: /sitemap.xml User-agent: Googlebot Disallow: /temp/ Allow: / Sitemap: https://example.com/sitemap.xml Sitemap: https://example.com/sitemap-articles.xml
Frequently Asked Questions
Find quick answers to common questions about plain text files, encodings, and online tools.
Upload your robots.txt file to the root directory of your web server so it is accessible at https://yourdomain.com/robots.txt.