AEO / AI Optimization
Crawl Delay Limits
What is this?
Strict crawl delays in robots.txt can stop aggressive AI crawlers fetching your content in time, shrinking your footprint in their data.
How Igris Radar detects it
The Igris Radar scanner automatically flags Crawl Delay Limits during its comprehensive audit of your web property. We simulate real-world crawlers and attack vectors to find issues before they impact your business.
Scan your site for this issue