LLM SEO and how AI models find and describe your business
Language models learn about a business in two different ways, and each is controlled differently. Most LLM SEO advice mixes them up.
Last updated
Quick answer
LLM SEO is the work of making sure large language models such as ChatGPT, Claude, Gemini and Perplexity can find accurate information about your business and describe it correctly. Models learn from training data collected in the past and from pages they retrieve live when someone asks a question. You control which AI crawlers can access your site through robots.txt, keep your business details consistent everywhere they appear, and earn mentions on sites the models trust.
Key takeaways
- Training crawlers and search crawlers are separate: blocking one doesn't block the other.
- Blocking Google-Extended doesn't affect Google Search, in Google's own words.
- User-triggered fetchers such as ChatGPT-User may not follow robots.txt.
- Google Search ignores llms.txt; it neither helps nor harms rankings.
What LLM SEO covers
This guide is about how AI models get information about your business and the controls you have over it. Being the quoted answer in a search result is covered in answer engine optimisation, and being cited in AI search answers in generative engine optimisation.
Two ways an AI model knows about a business
| Training data | Live retrieval | |
|---|---|---|
| When it happens | Before the model is released | When someone asks a question |
| Collected by | Training crawlers such as GPTBot and ClaudeBot | Search crawlers and user-triggered fetchers |
| Can it change quickly? | No: it reflects the past | Yes: it reads your current pages |
| What you control | Whether future training can use your content | Whether AI search can index and cite you |
For most businesses, live retrieval matters more. It's how an assistant answering "best SEO agency near me" or "who sells this product" reads your current site.
AI crawlers and what each one controls
| Crawler | Company | What it does | robots.txt token |
|---|---|---|---|
| GPTBot | OpenAI | Collects content that may be used to train its models | GPTBot |
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT's search features | OAI-SearchBot |
| ChatGPT-User | OpenAI | Fetches a page when a user asks; OpenAI says robots.txt rules may not apply | ChatGPT-User |
| ClaudeBot | Anthropic | Collects content for model training | ClaudeBot |
| Claude-SearchBot | Anthropic | Indexes content for Claude's search | Claude-SearchBot |
| Claude-User | Anthropic | Retrieves pages in response to a user's question | Claude-User |
| Google-Extended | Controls whether content is used to train and ground Gemini; not a separate crawler | Google-Extended | |
| PerplexityBot | Perplexity | Surfaces sites in Perplexity's results; Perplexity says it isn't used for model training | PerplexityBot |
| Perplexity-User | Perplexity | Fetches a page when a user asks; generally ignores robots.txt | Perplexity-User |
| CCBot | Common Crawl | Builds an open archive of the web | CCBot |
Google says Google-Extended doesn't affect a site's inclusion in Google Search and isn't a ranking signal. Blocking it is purely a choice about Gemini.
robots.txt: allow search, decide on training
- Keep the search crawlers allowedOAI-SearchBot, Claude-SearchBot and PerplexityBot are how AI search finds and cites you. Blocking them removes you from those answers.
- Decide on training separatelyGPTBot, ClaudeBot and Google-Extended govern future model training. Allowing or blocking them is a business choice, not an SEO one.
- Write one rule per crawlerA named group in robots.txt replaces the general rules for that crawler, so list each one you want to treat differently.
- Repeat it on every subdomainEach subdomain has its own robots.txt; Anthropic notes the block must be added on every subdomain you want to opt out.
- Allow time to take effectOpenAI says it can take about 24 hours after a robots.txt change for its systems to adjust.
For example, to stay in AI search but opt out of training, a robots.txt would disallow GPTBot, ClaudeBot and Google-Extended while leaving the search crawlers allowed. This site allows all of them, because being found in AI answers is the goal.
llms.txt: what it is and what Google says
llms.txt is a proposal for a Markdown file at the root of a site that gives language models a summary of its content. Its authors say it's mainly meant for use when a model answers a question, rather than for training.
Google is explicit that Search ignores llms.txt files: adding one will neither help nor harm a site's visibility in Google Search. Treat it as optional, not as a ranking lever.
Entity consistency: describe your business the same way everywhere
When sources disagree about your business, models repeat the confusion. Check that these match everywhere they appear: your website, Google Business Profile, LinkedIn, industry directories and review sites.
- Business name, written the same way each time.
- Address or service area.
- Phone number and email.
- The services you offer, in the same words.
- Founding year and who runs the business, if you publish them.
- Your website address in every profile.
Mentions on other sites
AI answers lean on what other sources say about a business, not only what it says about itself. Genuine coverage helps: industry association listings, supplier and partner pages, local media, and reviews on established platforms. Google warns that seeking inauthentic mentions across the web isn't as helpful as it might seem, so buying placements to game AI answers is wasted money.
What you can and can't influence
| You can | You can't |
|---|---|
| Choose which crawlers access your site | Pay or submit your site to appear in AI answers |
| Keep your business facts consistent | Change what a model learned in past training |
| Publish clear, accurate pages that answer questions | Control the exact wording an assistant uses |
| Earn genuine coverage on trusted sites | Guarantee a mention for any query |
Checking how AI models describe you
- Pick ten questionsThe ones a customer would ask, such as "who does [service] in [city]".
- Ask several assistantsChatGPT, Claude, Gemini and Perplexity, with web search on where available.
- Log the answersDate, assistant, whether you were mentioned, what was said and which sources were cited.
- Repeat monthlyAnswers vary between runs, so look for patterns over months, not single results.
- Check your server logsSearch for the crawler names in the table above to see which ones visit.
Common questions
Does blocking GPTBot remove my site from ChatGPT search?
No. OpenAI uses OAI-SearchBot to surface websites in ChatGPT's search features. GPTBot relates to training. Block one without the other if that's what you want.
Does blocking Google-Extended hurt my Google rankings?
No. Google says Google-Extended doesn't affect a site's inclusion in Google Search and isn't a ranking signal.
Do I need an llms.txt file?
No. Google Search ignores llms.txt, and it neither helps nor harms rankings there. It's an optional proposal some tools read.
How do ChatGPT and Claude get information about my business?
From training data collected before the model was released, and from pages their search crawlers and user-triggered fetchers retrieve when someone asks a question.
Can I pay to appear in AI answers?
Not in the organic answers. You can make accurate information easy to find, keep it consistent and earn genuine coverage, but no one can guarantee a mention.
Why does ChatGPT describe my business wrongly?
Usually because sources disagree or are out of date. Fix your own pages and profiles first, then the directories and listings that repeat old details.
Where to next
If you would rather have someone do this than read about it, the commercial page for this work is Technical SEO services.
All guides Generative engine optimisation (GEO)Answer engine optimisation (AEO)Semantic SEO