Italiano
English
Español
Français
DeutschWhy it matters
If a crawler cannot read a page — because it is blocked, because the content only arrives via JavaScript, or because the page is too heavy — that content will never make it into an answer.
In practice
- Check that robots.txt does not exclude them by mistake.
- Serve content in the HTML, not only after script execution.
- Keep pages light: a crawler that times out does not come straight back.
- Declare the structure of the site, including with an llms.txt file.
Related terms
Frequently asked questions
What is AI crawler?
An AI crawler is the program an artificial intelligence company uses to read web pages, either to train its models or to answer in real time. The most common are GPTBot, ClaudeBot, PerplexityBot and Google-Extended.
Why does AI crawler matter?
If a crawler cannot read a page — because it is blocked, because the content only arrives via JavaScript, or because the page is too heavy — that content will never make it into an answer.
Find out how your brand appears in AI answers today. The analysis takes about ten minutes.