Free Robots.txt Validator & Sitemap Checker

Free Robots.txt Checker

Verify if a website's robots.txt file exists, inspect the live file on screen, validate Sitemap: URL declarations, and get an instant SEO & AI crawler rating.

webstix.com • https://webstix.com/robots.txt ↗ • Checked 2026-10-16 09:50:57
Rescan Live XML Sitemap Audit → Noindex Checker →
100 / 100
Grade A+
Robots.txt Rating
Healthy & Optimized
10-Point Crawler Audit
File Existence Status
Found (HTTP 200)
316 bytes • 18ms response
Sitemap URL Presence
Present (1)
Declared inside robots.txt
Rule Inventory
10 Lines
1 Bot Groups • 6 Disallow • 1 Allow

Healthy & Optimized

Your robots.txt file is live, properly formatted, allows search engine indexing, and declares your XML Sitemap.

Live robots.txt File Viewer (webstix.com)

HTTP 200 OK • 10 Lines
https://webstix.com/robots.txt ↗
1 User-agent: *
2 Disallow: /wp-content/uploads/wc-logs/
3 Disallow: /wp-content/uploads/woocommerce_transient_files/
4 Disallow: /wp-content/uploads/woocommerce_uploads/
5 Disallow: /*?add-to-cart=
6 Disallow: /*?*add-to-cart=
7 Disallow: /wp-admin/
8 Allow: /wp-admin/admin-ajax.php
9
10 Sitemap: https://webstix.com/sitemap_index.xml

Sitemap URL Presence & Reachability Check

1 Sitemap URL Declared
Line Declared Sitemap URL HTTP Status Protocol Response Time Deep Audit
L10 https://webstix.com/sitemap_index.xml ↗ HTTP 200 OK HTTPS 2291ms Audit Sitemap →

Content Rating & Best-Practice Audit Checklist

Score: 100 / 100 (Grade A+)
Robots.txt File Existence & HTTP 200 OK
Found valid plain-text robots.txt at https://webstix.com/robots.txt (HTTP 200 OK in 18ms).
Passed
User-Agent Rule Groups
Configured with 1 rule group(s) across 1 User-agent declaration(s).
Passed
Homepage & Root Crawlability
Googlebot, Bingbot, and standard search crawlers are permitted to crawl your root domain ("/").
Passed
XML Sitemap URL Declaration
Declares 1 XML Sitemap URL inside robots.txt.
Passed
Declared Sitemap Accessibility
All verified Sitemap URL(s) use HTTPS and responded with HTTP 200 OK.
Passed
RFC 9309 Syntax Validation
Zero syntax errors detected across 10 line(s).
Passed
Unsupported / Deprecated Directives
Does not rely on retired robots.txt Noindex directives.
Passed
CSS & JavaScript Rendering Access
CSS stylesheets and JavaScript bundles are not blocked by Disallow rules.
Passed
Crawl-Delay Throttling Check
No restrictive Crawl-delay throttling applied.
Passed
File Size & Parser Limit (500 KB Max)
Lightweight file size (316 bytes / 0.31 KB) — well within Google's 500 KB limit.
Passed
AI Search & LLM Crawler Permissions
All 5 major AI search & training crawlers (OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended) are permitted.
Info

Interactive URL Path & Bot Simulator

Test any path against webstix.com's live rules

Search Engine & AI Crawler Access Matrix

Evaluated against root path /
Crawler / User-Agent Category Access Status Matched Group Effective Rule
Googlebot
Google Web Search
 Search Engine ✓ Allowed User-agent: * Implicit Allow
Bingbot
Microsoft Bing Search
 Search Engine ✓ Allowed User-agent: * Implicit Allow
DuckDuckBot
DuckDuckGo Search
 Search Engine ✓ Allowed User-agent: * Implicit Allow
Applebot
Apple Siri & Spotlight
 Search Engine ✓ Allowed User-agent: * Implicit Allow
OAI-SearchBot
ChatGPT Live Search
 AI Search ✓ Allowed User-agent: * Implicit Allow
GPTBot
OpenAI Model Training
 AI Training ✓ Allowed User-agent: * Implicit Allow
ClaudeBot
Anthropic Claude AI
 AI Assistant ✓ Allowed User-agent: * Implicit Allow
PerplexityBot
Perplexity Answer Engine
 AI Search ✓ Allowed User-agent: * Implicit Allow
Google-Extended
Google Gemini Training
 AI Training ✓ Allowed User-agent: * Implicit Allow

Recent robots.txt Audits

example.com
Missing • Sitemap: No
15/100 (F)
webstix.com
Found (HTTP 200) • Sitemap: Yes
100/100 (A+)

What Makes a High-Rating robots.txt File?

1. Always Declare Your XML Sitemap

Search engine crawlers read /robots.txt before anything else on your domain. Including Sitemap: https://yourdomain.com/sitemap.xml guarantees that Googlebot, Bingbot, and AI search engines find all of your indexable pages immediately.

2. Never Block CSS or JavaScript

Google renders pages using a headless Chromium browser. If your robots.txt blocks *.css, *.js, or /wp-includes/, Googlebot cannot verify mobile responsiveness and may downgrade your rankings.

3. Use Robots.txt for Crawl Budget, Not De-Indexing

Blocking a URL with Disallow: prevents bots from crawling the page, but Google can still index the URL if external links point to it. To remove a page from search results, allow crawling and use a <meta name="robots" content="noindex"> tag instead.