Search & AI visibility monitors
A visibility monitor checks that search engines and AI search tools can still find and index a page. A single deploy can add a noindex tag, block crawlers in robots.txt or empty the sitemap. Rankings then drop days later, and nobody notices the cause.
Each check fetches:
- the page itself, following redirects
robots.txton the page's final host- the sitemap listed in
robots.txt, or/sitemap.xmlif none is listed. Up to 10 child sitemaps of a sitemap index are read, and gzipped sitemaps are supported. llms.txt, if you turn that rule on
Settings
| Setting | Default | Notes |
|---|---|---|
| URL | None | Usually your home page or an important landing page. |
| Interval | 1 hour | 15 minutes, 30 minutes, 1 hour, 6 hours or 1 day. |
| Page is marked noindex | On | Fails on noindex or none in a robots meta tag or an X-Robots-Tag header. This covers the generic tag and tags aimed at one of the required crawlers. |
| Sitemap is missing or empty | On | Fails when the sitemap does not load, is not a valid sitemap or lists no URLs. |
| Canonical points to another host | On | Fails when rel=canonical (in the page or a Link header) points to a different host, such as a staging domain. |
| llms.txt is missing | Off | Fails when /llms.txt does not load, is empty or returns an HTML page. |
| Title, canonical or robots.txt change | On | Compares each check with a baseline. See Change tracking. |
| Crawlers that must be allowed | Search and AI search crawlers | Fails when robots.txt blocks a selected crawler from the page. |
| Sitemap drop | 20 % | With change tracking on, fails when the sitemap lists this share of URLs fewer than at the baseline. |
Crawlers
| Crawler | Used by | Required by default |
|---|---|---|
| Googlebot | Google Search | Yes |
| Bingbot | Bing | Yes |
| OAI-SearchBot | ChatGPT search | Yes |
| Claude-SearchBot | Claude search | Yes |
| PerplexityBot | Perplexity | Yes |
| GPTBot | OpenAI model training | No |
| ClaudeBot | Anthropic model training | No |
| Google-Extended | Gemini model training | No |
Training crawlers are not required by default. Many sites block them on purpose and still want to appear in AI search answers. The monitor page shows the access of every crawler either way.
Change tracking
The first successful check saves a baseline: the final URL, title, canonical, structured data types, which crawlers robots.txt allows, and the sitemap URL count. Later checks fail when:
- the page now redirects somewhere else
- the title or canonical changes or disappears
- the page becomes
noindex - structured data types (JSON-LD
@type) disappear robots.txtnewly blocks a crawler- the sitemap shrinks by the configured percentage
A changed robots.txt that still allows every crawler is shown as a note and does not alert.
If a change was intended, open the monitor and choose Accept current state as baseline. The monitor checks again right away and closes the incident. Changing the monitor's URL also starts a new baseline.
When the monitor fails
| Reason shown | Meaning |
|---|---|
Page is marked noindex via <meta name="robots"> |
The page asks search engines not to index it. |
Page is marked noindex via X-Robots-Tag header |
The server sends a noindex header, often from a CDN rule or a framework's preview setting. |
robots.txt blocks Googlebot from / |
A required crawler is disallowed for this page's path. |
robots.txt is unreachable |
robots.txt returned a 5xx, 429 or a connection error. Google then treats the whole site as disallowed. A 404 counts as "allow everything" and is fine. |
Sitemap https://… returned 404 |
The sitemap, or one of the child sitemaps in an index, did not load. |
Canonical URL points to another host |
The page names a different host as the original, so search engines index that host instead. |
Title changed to "…" (was "…") |
Change tracking spotted a difference from the baseline. |
Sitemap URL count dropped from 120 to 40 (-67%) |
Pages were removed from the sitemap. |
A failed check is retried once before an incident opens, just like other monitors. Visibility problems don't depend on location, so remote probes are not asked to confirm them.
Good to know
- Visibility monitors count toward your uptime monitor limit.
- They can't be added to public status pages, because they report search setup rather than downtime.
- Checks use the
AsniherBotuser agent. A page that serves different content to Googlebot will look the same to us as to a normal browser.