If you want to know why ChatGPT keeps ignoring your beautifully designed pricing page while citing your competitor's ugly, text-heavy site, you need to understand one basic truth: AI bots don't "see" your website. They read your code. And frankly, they aren't very good at it.
Unlike the massively sophisticated Googlebot, which has spent two decades learning how to render complex JavaScript and execute cascading style sheets, modern AI crawlers like GPTBot (OpenAI), ClaudeBot (Anthropic), and PerplexityBot are comparatively primitive. They are built for one thing only: extracting raw text as quickly as possible.
The JavaScript Problem
Here's the biggest hurdle most modern web apps face when dealing with AI crawlers: client-side rendering. If your entire website is a single-page application built in React or Vue, and the server just sends down an empty HTML shell with a massive JavaScript bundle attached, AI bots will often just see a blank page.
While Google will patiently wait, download the bundle, execute the scripts, and render the final DOM tree, most AI agents simply do not have the compute budget for that. They grab the initial HTML payload and immediately move on. If your core content isn't in that initial payload, you effectively don't exist to them.
Igris Radar Data: What the Bot Actually Sees
When our scanner simulates a GPTBot crawl on a modern React site, here is what the raw extracted payload looks like. Notice how it strips all styling and relies entirely on semantic tags:
<H2> Pro Tier
<TEXT> $99/month for up to 5 domains.
<DROPPED> [SVG Icon]
<DROPPED> [CSS Animation class="fade-in"]
<DROPPED> [JavaScript tooltip showing feature details]
The Fix: If your feature details only exist inside a JS-powered hover tooltip, the AI never reads them. You must use Server-Side Rendering (SSR) and put critical text directly in the DOM.
What AI Crawlers Actually Care About
Once an AI bot has the HTML, it runs a brutal stripping process. It strips away the CSS. It drops the interactive scripts. It removes the navigation menus and footer boilerplate. It is hunting exclusively for semantic structure and text.
1. Semantic Headings (H1, H2, H3)
To an LLM, headings are the map of your content. If you use a massive, bolded <span> tag because it looks cool, the AI crawler just sees regular text. You must use proper semantic headings. The bot uses these H-tags to determine the context of the paragraphs that follow them.
2. Clean, Formatted Tables
If you are comparing two products, do not use complex flexbox grids or CSS layouts to build your comparison chart. AI crawlers struggle to understand spatial relationships defined in CSS. Instead, use boring, traditional <table> tags. Language models parse raw HTML tables incredibly well, easily mapping rows to columns to understand the structured data.
3. Schema.org (JSON-LD)
Crawlers love structured data. If you bury your author's credentials in a quirky bio at the bottom of the page, the bot might miss it. If you explicitly declare it in a JSON-LD script block using the Person schema, the bot ingests it instantly with zero ambiguity. The same goes for product pricing, reviews, and FAQ sections.
The "Ignore" List
It's equally important to know what these bots completely ignore. Don't waste time optimizing these elements for Answer Engines:
- Images and Alt Text: While crucial for accessibility and traditional Google Image search, most text-based LLM crawlers completely drop
<img>tags during the parsing phase. Don't put vital facts exclusively in infographics. - Accordion Menus: If the content inside an accordion or tabbed interface relies on JavaScript to be loaded into the DOM, the bot won't click the button to reveal it. Make sure the text is present in the source HTML, even if CSS hides it visually.
Checking Your Own Site
The best way to see your site through the eyes of a crawler is to disable JavaScript in your browser and view the raw source. Does it make sense? Are the headings clear? Is the text immediately accessible?
If you want a more comprehensive analysis, the Igris Radar AEO Audit specifically mimics the behavior of these AI agents. We strip the payload exactly how they do and score your remaining semantic structure, giving you a clear roadmap for fixing your extraction issues.