Free Robots.txt Validator & Sitemap Checker

Free Robots.txt Checker

Verify if a website's robots.txt file exists, inspect the live file on screen, validate Sitemap: URL declarations, and get an instant SEO & AI crawler rating.

example.com • https://example.com/robots.txt ↗ • Checked 2026-10-16 10:31:18
Rescan Live XML Sitemap Audit → Noindex Checker →
15 / 100
Grade F
Robots.txt Rating
Missing (HTTP 404)
10-Point Crawler Audit
File Existence Status
Missing (HTTP 404)
HTML Soft-404 returned instead of text
Sitemap URL Presence
Not Declared
No Sitemap: directive found in file
Rule Inventory
0 Lines
0 Bot Groups • 0 Disallow • 0 Allow

Missing (HTTP 404)

We could not find a valid plain-text robots.txt file at https://example.com/robots.txt. Creating one with a Sitemap declaration is strongly recommended.

Live robots.txt File Viewer (example.com)

File Missing / Unreachable

No Valid robots.txt File Found at https://example.com/robots.txt

The server responded with HTTP 404 (returned an HTML page instead of plain text). Below is a clean, starter robots.txt template you can copy or download and upload to the root of example.com:

Recommended starter /robots.txt for example.com 
# Optimized robots.txt for example.com
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap.xml

Sitemap URL Presence & Reachability Check

Missing Sitemap Directive
No Sitemap: URL is declared inside this robots.txt file.

Adding Sitemap: https://example.com/sitemap.xml to the bottom of your robots.txt file helps Googlebot, Bingbot, and AI search crawlers discover new and updated pages faster.

Content Rating & Best-Practice Audit Checklist

Score: 15 / 100 (Grade F)
Robots.txt File Format & Availability
Your server returned an HTML webpage instead of a plain-text robots.txt file at https://example.com/robots.txt (Soft-404 / misconfigured routing).
Critical
XML Sitemap Declaration
No robots.txt file is present to advertise your XML Sitemap URL to search crawlers.
Critical

Search Engine & AI Crawler Access Matrix

Evaluated against root path /
Crawler / User-Agent Category Access Status Matched Group Effective Rule
Googlebot
Google Web Search
 Search Engine ✓ Allowed User-agent: * No robots.txt (Implicit Allow)
Bingbot
Microsoft Bing Search
 Search Engine ✓ Allowed User-agent: * No robots.txt (Implicit Allow)
DuckDuckBot
DuckDuckGo Search
 Search Engine ✓ Allowed User-agent: * No robots.txt (Implicit Allow)
Applebot
Apple Siri & Spotlight
 Search Engine ✓ Allowed User-agent: * No robots.txt (Implicit Allow)
OAI-SearchBot
ChatGPT Live Search
 AI Search ✓ Allowed User-agent: * No robots.txt (Implicit Allow)
GPTBot
OpenAI Model Training
 AI Training ✓ Allowed User-agent: * No robots.txt (Implicit Allow)
ClaudeBot
Anthropic Claude AI
 AI Assistant ✓ Allowed User-agent: * No robots.txt (Implicit Allow)
PerplexityBot
Perplexity Answer Engine
 AI Search ✓ Allowed User-agent: * No robots.txt (Implicit Allow)
Google-Extended
Google Gemini Training
 AI Training ✓ Allowed User-agent: * No robots.txt (Implicit Allow)

How to Improve Your robots.txt Rating

[Server Configuration] Replace HTML Soft-404 with a Plain-Text robots.txt File
Search engine crawlers expect /robots.txt to return Content-Type: text/plain. Returning HTML markup causes crawlers to ignore your directives or log syntax errors in Google Search Console.

Recent robots.txt Audits

example.com
Missing • Sitemap: No
15/100 (F)
webstix.com
Found (HTTP 200) • Sitemap: Yes
100/100 (A+)

What Makes a High-Rating robots.txt File?

1. Always Declare Your XML Sitemap

Search engine crawlers read /robots.txt before anything else on your domain. Including Sitemap: https://yourdomain.com/sitemap.xml guarantees that Googlebot, Bingbot, and AI search engines find all of your indexable pages immediately.

2. Never Block CSS or JavaScript

Google renders pages using a headless Chromium browser. If your robots.txt blocks *.css, *.js, or /wp-includes/, Googlebot cannot verify mobile responsiveness and may downgrade your rankings.

3. Use Robots.txt for Crawl Budget, Not De-Indexing

Blocking a URL with Disallow: prevents bots from crawling the page, but Google can still index the URL if external links point to it. To remove a page from search results, allow crawling and use a <meta name="robots" content="noindex"> tag instead.