Documentation menu

Search & AI visibility monitors

A visibility monitor checks that search engines and AI search tools can still find and index a page. A single deploy can add a noindex tag, block crawlers in robots.txt or empty the sitemap. Rankings then drop days later, and nobody notices the cause.

Each check fetches:

  • the page itself, following redirects
  • robots.txt on the page's final host
  • the sitemap listed in robots.txt, or /sitemap.xml if none is listed. Up to 10 child sitemaps of a sitemap index are read, and gzipped sitemaps are supported.
  • llms.txt, if you turn that rule on

Settings

Setting Default Notes
URL None Usually your home page or an important landing page.
Interval 1 hour 15 minutes, 30 minutes, 1 hour, 6 hours or 1 day.
Page is marked noindex On Fails on noindex or none in a robots meta tag or an X-Robots-Tag header. This covers the generic tag and tags aimed at one of the required crawlers.
Sitemap is missing or empty On Fails when the sitemap does not load, is not a valid sitemap or lists no URLs.
Canonical points to another host On Fails when rel=canonical (in the page or a Link header) points to a different host, such as a staging domain.
llms.txt is missing Off Fails when /llms.txt does not load, is empty or returns an HTML page.
Title, canonical or robots.txt change On Compares each check with a baseline. See Change tracking.
Crawlers that must be allowed Search and AI search crawlers Fails when robots.txt blocks a selected crawler from the page.
Sitemap drop 20 % With change tracking on, fails when the sitemap lists this share of URLs fewer than at the baseline.

Crawlers

Crawler Used by Required by default
Googlebot Google Search Yes
Bingbot Bing Yes
OAI-SearchBot ChatGPT search Yes
Claude-SearchBot Claude search Yes
PerplexityBot Perplexity Yes
GPTBot OpenAI model training No
ClaudeBot Anthropic model training No
Google-Extended Gemini model training No

Training crawlers are not required by default. Many sites block them on purpose and still want to appear in AI search answers. The monitor page shows the access of every crawler either way.

Change tracking

The first successful check saves a baseline: the final URL, title, canonical, structured data types, which crawlers robots.txt allows, and the sitemap URL count. Later checks fail when:

  • the page now redirects somewhere else
  • the title or canonical changes or disappears
  • the page becomes noindex
  • structured data types (JSON-LD @type) disappear
  • robots.txt newly blocks a crawler
  • the sitemap shrinks by the configured percentage

A changed robots.txt that still allows every crawler is shown as a note and does not alert.

If a change was intended, open the monitor and choose Accept current state as baseline. The monitor checks again right away and closes the incident. Changing the monitor's URL also starts a new baseline.

When the monitor fails

Reason shown Meaning
Page is marked noindex via <meta name="robots"> The page asks search engines not to index it.
Page is marked noindex via X-Robots-Tag header The server sends a noindex header, often from a CDN rule or a framework's preview setting.
robots.txt blocks Googlebot from / A required crawler is disallowed for this page's path.
robots.txt is unreachable robots.txt returned a 5xx, 429 or a connection error. Google then treats the whole site as disallowed. A 404 counts as "allow everything" and is fine.
Sitemap https://… returned 404 The sitemap, or one of the child sitemaps in an index, did not load.
Canonical URL points to another host The page names a different host as the original, so search engines index that host instead.
Title changed to "…" (was "…") Change tracking spotted a difference from the baseline.
Sitemap URL count dropped from 120 to 40 (-67%) Pages were removed from the sitemap.

A failed check is retried once before an incident opens, just like other monitors. Visibility problems don't depend on location, so remote probes are not asked to confirm them.

Good to know

  • Visibility monitors count toward your uptime monitor limit.
  • They can't be added to public status pages, because they report search setup rather than downtime.
  • Checks use the AsniherBot user agent. A page that serves different content to Googlebot will look the same to us as to a normal browser.

Questions about this page? Email support@asniher.com.