Scanner Rule
robots.txt configuration
What is this?
The robots.txt file tells search-engine crawlers which pages they may and may not crawl (the Robots Exclusion Protocol).
How Igris Radar detects it
The Igris Radar scanner automatically flags robots.txt configuration during its comprehensive audit of your web property. We simulate real-world crawlers and attack vectors to find issues before they impact your business.
Scan your site for this issue