Can I customize this template before downloading?

Yes! You can edit the text directly in the interactive workspace above, change line endings between Windows CRLF and Unix LF, and download your customized version with 1 click.

Back to Templates Library

Template / WEBMASTER

Standard Webmaster robots.txt Template

Universal crawler control file following the Robots Exclusion Standard to guide Googlebot, Bingbot, and protect private routes.

14px

# ====================================================================
# Robots Exclusion Protocol (robots.txt)
# Website: https://www.example.com
# Generated: 2026-08-26
# ====================================================================

# Rule 1: Allow all reputable search engines full access to public pages
User-agent: *
Disallow: /admin/
Disallow: /dashboard/
Disallow: /api/private/
Disallow: /checkout/
Disallow: /cart/
Disallow: /account/
Disallow: /search?*
Disallow: /*.json$
Disallow: /*.xml$
Allow: /sitemap.xml
Allow: /assets/
Allow: /static/

# Rule 2: Explicit instructions for Googlebot
User-agent: Googlebot
Disallow: /admin/
Disallow: /temp/
Allow: /

# Rule 3: Explicit instructions for Bingbot
User-agent: Bingbot
Disallow: /admin/
Crawl-delay: 1

# Rule 4: Block aggressive AI scrapers (Optional)
User-agent: GPTBot
Disallow: /private-docs/

User-agent: CCBot
Disallow: /private-docs/

# XML Sitemap Location (Crucial for SEO discovery)
Sitemap: https://www.example.com/sitemap.xml
Sitemap: https://www.example.com/sitemap-articles.xml

1,061 chars(946 no spaces)

112 words

42 lines(7 paras)

~1 min read

Size: 1.04 KB

LFutf-8College Graduate (Very Difficult)

Tags:#seo#robots#webmaster#googlebot#crawlers

Frequently Asked Questions

Find quick answers to common questions about plain text files, encodings, and online tools.

This template is typically saved as robots.txt using UTF-8 character encoding.