Why AI search cites some local businesses and ignores others: schema, llms.txt and crawler access

Owners ask us a new question this year. Someone typed "good electrician in Tallinn" into ChatGPT, got three names back, and theirs was not one of them. Why?

Nobody outside OpenAI, Anthropic or Google can see how an assistant picks those three names, and anyone who claims otherwise is selling something. What we can measure is the step before that: whether the machines are allowed to read your site at all, and whether what they read says plainly who you are, where you work and how to reach you.

So we measured it. On 18 October 2026 we took the Google top 10 for 20 service searches in Tallinn, from "elektrik tallinn" and "lukksepp tallinn" to "hambaarst tallinn" and "rehvivahetus tallinn". We removed directories, social networks and marketplaces, kept 105 independent business sites and scanned the 100 that answered.

What the AI companies themselves say

Before the numbers, the rules. They are public, and they are more specific than most advice going around.

OpenAI runs separate crawlers. OAI-SearchBot is the one used for ChatGPT search, and OpenAI states that a site which blocks it will not be shown in ChatGPT search answers. GPTBot collects training data, and blocking it is a separate decision. OpenAI's documentation says each setting is independent of the others.

Anthropic splits it the same way. ClaudeBot is for training. Claude-SearchBot indexes pages for search, and Anthropic says blocking it may reduce your site's visibility and accuracy in user search results. Claude-User fetches a page when a person asks Claude about it.

Google is the most direct of the three. Its page on AI Overviews and AI Mode says there are no additional requirements to appear there, no special schema.org markup to add, and no new AI text files to create. What it does say is that structured data must match what is visible on the page.

That leaves three things a business controls: whether the crawlers get in, whether the site states its facts in a form a machine cannot misread, and whether those facts are visible on the page in the first place.

Seven sites send the search crawlers away and do not know it

Every one of the 96 robots.txt files we could read allowed the AI crawlers. On paper, nobody blocks anything.

The server told a different story. Seven sites answered 403 Forbidden to GPTBot, ClaudeBot, Claude-SearchBot and CCBot, while OAI-SearchBot, PerplexityBot, ChatGPT-User and Claude-User received a normal page. All seven sat on the same Estonian shared hosting. An eighth site on that very server let every bot through, so this is a filter switched on per hosting account, not something the owner wrote into robots.txt.

For ChatGPT search these sites are fine: OAI-SearchBot gets in. For Claude's search index they are closed, and by Anthropic's own wording that can cost visibility. None of it shows up in robots.txt, Search Console or the site's admin panel. You only see it if you request the page as the bot.

One of the seven also refused requests that identified as Googlebot, AdsBot-Google and Bingbot. Our requests did not come from Google's or Microsoft's addresses, and some firewalls block fake search bots on purpose, so this alone does not prove Google is locked out. It is exactly the kind of thing to confirm in Search Console's URL Inspection before a month of traffic goes missing.

Eighty sites have structured data. Twenty-five say what the business is.

Structured data is how a page states facts in a form software does not have to interpret: this is a business, this is its address, these are its hours. Google does not require it for AI features, and nobody can promise that an assistant reads it. It is still the least ambiguous way to say who you are.

Eighty of the hundred homepages carried some JSON-LD. Most of it was what an SEO plugin produces by default: WebSite on 69 sites, a site search box on 60, a generic Organization on 56. Only 25 sites described themselves as a LocalBusiness or one of its subtypes. Forty-three stopped at Organization, which says "a company exists" and nothing about where it works or when it is open.

Field inside the 25 LocalBusiness blocks	Sites
Telephone	25
Address	24
Opening hours	20
Links to other profiles (sameAs)	16
Price range	15
Area served	14
Map coordinates	12

The sites that bothered to describe a local business mostly did it properly. The gap is between those 25 and the other 75. If an assistant has to decide whether you are a plumber who works in Lasnamäe on Saturdays, a page that says so explicitly, in visible text and in markup that matches it, leaves less room for a wrong guess.

llms.txt: fifteen real files, ten fake ones

llms.txt is a proposal from 2024: a Markdown file at the root of the site that gives language models a short summary and a list of the pages worth reading. It is not a standard, and Google says outright that you don't need it for its AI features.

Fifteen of the hundred sites had one. The more interesting number is ten: ten sites answered /llms.txt with an ordinary HTML page and status 200. The CMS serves its homepage or a "page not found" template without the 404 code. A file that does not exist looks, to any crawler, like a page that does.

That is a hygiene problem on its own, and it is not about AI. A site that answers 200 to addresses that do not exist does the same for every broken link and every old URL.

The facts an assistant would need are often missing from the page

Ask a person to recommend a locksmith and they want to know three things: does this firm do the job, is it in my area, how do I reach it. Often a fourth: roughly what does it cost.

On 39 of the 100 homepages there was no tap-to-call phone link. On 26 there was neither a phone link nor an email link. On 69 there was not a single euro figure anywhere in the visible text. We did not measure what assistants answer, but it is plain arithmetic that a price which is not published anywhere cannot be quoted.

What we would fix, in this order

1. Request your homepage as OAI-SearchBot, Claude-SearchBot and Googlebot and check that each gets a 200. Ten minutes, and it is the only item here that can quietly remove you from an index.
2. Put the plain facts in visible text: services, districts or towns served, phone, hours, and at least a "from" price.
3. Describe the business with the most specific LocalBusiness subtype that fits, with the same name, address, phone and hours as the visible page.
4. Link the markup to your Google Business Profile and other real profiles with sameAs, so all of them read as one business.
5. Add llms.txt last, and make sure addresses that do not exist return a real 404.

The step-by-step version with copy-paste templates and test commands is in our playbook on schema, llms.txt and AI crawler checks.

How we counted

Search results came from Google Estonia, Estonian language, desktop, top 10, collected on 18 October 2026. Each homepage was fetched once with an ordinary browser user agent and once per crawler. Bots were tested by their user-agent string from our server. Rate-limit responses (429) were not counted as blocks, only 403. We did not test what any assistant answers, and this article does not claim that schema or llms.txt makes one cite you. It shows what the machines can and cannot read.

Common questions

Does structured data make ChatGPT recommend my business?

No one can promise that, and Google says its AI features need no special markup. Structured data removes ambiguity about who you are, where you work and how to reach you. It helps only when the same facts are visible on the page.

Should I block GPTBot and ClaudeBot?

That is a decision about training data, and it is yours to make. Blocking GPTBot does not remove you from ChatGPT search, because that uses OAI-SearchBot. Blocking Claude-SearchBot is different: Anthropic says it may reduce your visibility in Claude's search results.

Do I need an llms.txt file?

It is optional. llms.txt is a proposal, not a standard, and Google does not use it for AI Overviews. It is cheap to add, but it belongs at the end of the list, after crawler access and correct business facts.

How do I know if my hosting blocks AI crawlers?

Request your homepage with the crawler's user agent and look at the status code. A 403 for Claude-SearchBot or OAI-SearchBot while normal visitors get 200 means the server filters them, even when robots.txt allows everything.