Secrets & Exposure
low Severity
Free Check
robots.txt Sensitive Path Exposure
Audits robots.txt to ensure Disallow: directives do not inadvertently advertise secret administrative or staging URLs.
Why It Matters
Attackers inspect robots.txt first to find hidden administrative routes, unreleased features, or sensitive paths.
How We Check This
We fetch /robots.txt and analyze Disallow paths for sensitive naming patterns (e.g. /private/, /backup/, /staging/).
How to Fix & Implement
Copy-paste configuration blocks tailored for your web server or edge proxy.
nginx snippet
# Only list public indexing rules in robots.txt. Enforce authentication on sensitive endpoints rather than relying on robots.txt
Frequently Asked Questions
Does robots.txt protect pages from being accessed?
No! robots.txt only requests search engine crawlers not to index a page. Anyone can still open the URL.