Secrets & Exposure
low Severity
Free Check

robots.txt Sensitive Path Exposure

Audits robots.txt to ensure Disallow: directives do not inadvertently advertise secret administrative or staging URLs.

Test Your Website for robots.txt Sensitive Path Exposure

Run this check along with 90+ automated tests across your domain.

Why It Matters

Attackers inspect robots.txt first to find hidden administrative routes, unreleased features, or sensitive paths.

How We Check This

We fetch /robots.txt and analyze Disallow paths for sensitive naming patterns (e.g. /private/, /backup/, /staging/).

How to Fix & Implement

Copy-paste configuration blocks tailored for your web server or edge proxy.

nginx snippet
# Only list public indexing rules in robots.txt. Enforce authentication on sensitive endpoints rather than relying on robots.txt

Frequently Asked Questions

Does robots.txt protect pages from being accessed?

No! robots.txt only requests search engine crawlers not to index a page. Anyone can still open the URL.