How to Get Your Website Crawled and Cited by ChatGPT, Perplexity, and AI Search in 2026

"Does ChatGPT even see my website" is a fair question, and the honest answer is: it depends which ChatGPT you mean. There is a real, actionable difference between the crawlers that train AI models and the ones that fetch a live answer for a user's question right now. Getting this right is mostly a robots.txt and structured-data exercise, not magic.

Two Different Kinds of AI Crawlers

Every major AI company runs at least two separate bots, and they do different jobs:

• Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) scrape content to help train or fine-tune a model. Blocking these does not stop you from being cited, it only stops that specific crawl from becoming training data.
• Retrieval or search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot-Extended) fetch a page in real time to answer a specific user question. Blocking these is what actually makes you invisible to live AI answers and citations.

Most sites that want visibility in AI answers, including this one, allow the retrieval crawlers and make a deliberate choice about the training crawlers rather than blocking everything by default.

Writing a robots.txt That Actually Controls This

A basic, honest starting point looks like this:

• User-agent: * with Allow: / as the default, so nothing gets blocked by accident.
• A Sitemap: line pointing to your sitemap, so crawlers (AI or otherwise) can find everything without guessing.
• A Content-Signal line, an emerging robots.txt extension backed by Cloudflare, that expresses your preference in three parts: search=yes (allow indexing and short excerpts), ai-input=yes or no (allow real-time use in generated answers), and ai-train=yes or no (allow use as model training data).

Worth knowing before you rely on it: Content-Signal is new, and not every crawler honors it yet. Treat it as a clearly stated preference layered on top of, not a replacement for, explicit per-bot Allow and Disallow rules if you want to guarantee a specific bot's behavior. Cloudflare itself changed its default policy on September 15, 2026, blocking training and agent crawlers by default on ad-supported pages while continuing to allow search crawlers by default, which is a good signal of where the industry consensus is heading.

Why Bing Webmaster Tools Matters More Than You Think

This is the piece most sites skip. ChatGPT's live search mode pulls its web results primarily from Bing's index, not Google's. If Bing has not crawled and indexed a page, ChatGPT's browsing and citation features generally cannot surface it, regardless of your Google ranking. Setting up Bing Webmaster Tools and submitting your sitemap there is a five-minute task that directly affects whether you can be cited at all. As of early 2026, Bing Webmaster Tools also ships an AI Performance Report showing how often Copilot and partner AI experiences are actually citing your pages, which is the closest thing to real measurement currently available for this.

The Honest Truth About llms.txt

llms.txt (a proposed plain-text summary file at the root of a domain) gets recommended constantly as an "AI SEO" fix, and it deserves a straight answer: Google has confirmed no system in Google Search reads or acts on it, comparing it to the old keywords meta tag, a self-reported signal search engines learned to ignore because site owners control it. Independent research analyzing over 130,000 sites found the overwhelming majority of llms.txt files receive effectively no engagement. No major AI provider has confirmed using it for citation or training decisions either.

That does not make it worthless, it makes it narrow. The one place it demonstrably helps is AI coding assistants, tools like Cursor, GitHub Copilot, and Claude that fetch a project's own documentation in real time and benefit from a clean, low-token summary of what is available. If you publish developer docs, it is worth having. If you are hoping it moves your ChatGPT citation numbers, the current evidence says it will not, and the structured data and Bing indexing work above will do far more.

Structure Content So an LLM Can Actually Use It

Once a crawler can reach your page, the content itself needs to be extractable, not just readable. A few things that consistently help:

• Short, self-contained paragraphs. Keep most paragraphs under roughly 70 words, and name the subject explicitly instead of leaning on "it" or "this" across sentences. Long context windows still lose track of an ambiguous pronoun.
• FAQPage and Article structured data together. Stacking schema types gives an LLM (and traditional search) more than one clean entry point into the same content.
• A direct answer before the explanation. State the conclusion first, then support it. Both AEO snippets and GEO citations favor content that leads with the answer rather than building up to it.
• Real specifics. Named numbers, dates, and sources are what get quoted. Vague, hedge-everything language rarely does.

A Working Checklist

• Confirm your robots.txt allows retrieval bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) and makes a deliberate choice about training bots
• Add a Content-Signal line alongside explicit bot rules, not instead of them
• Set up Bing Webmaster Tools and submit your sitemap, since this is the actual path into ChatGPT's live citations
• Keep llms.txt if you have developer docs, but do not count on it for citation visibility
• Mark up genuine FAQ content with FAQPage schema and pair it with Article schema on blog and guide pages
• Write direct-answer-first paragraphs with real specifics, not hedged generalities

We handle exactly this kind of setup, robots.txt, structured data, Bing indexing, and site architecture, as part of how we build and maintain sites. If you want your site checked against this list, get in touch or see how we approach technical strategy and consulting.