What a search engine and a language model can read from your site

A lot gets written about content. Not much gets checked about whether that content can be read at all. The second part takes ten minutes, and it is annoyingly often the reason a good text does nothing for you.

Three things, and you can check all three yourself.

1. Does every piece have its own address?

A PDF on an overview page is not an article. It gets no title of its own in the search results, it cannot carry structured data, and the crawlers that language models use read it worse than plain HTML.

Open your news page and click a post. If the address in the bar changes to something like /news/maintaining-natural-stone/, you are fine. If you get a download, or you stay on the same page, then for a search engine that piece does not exist as a page.

2. Is your text in the HTML, or does it arrive with JavaScript?

Google usually runs JavaScript. The crawlers that language models fetch pages with usually do not. If your text only appears after the page has assembled itself, you are visible to one and invisible to the other.

You can check it without any tooling: open the page, choose View Page Source, and search for a sentence from your own text. If it is not there, it came from JavaScript.

3. Does your sitemap know which domain you are on?

A sitemap is the list you hand over yourself. If it holds a different domain than the one your site runs on, for instance because you changed names at some point, you are handing over a list of addresses that all redirect. Not fatal, but your live domain effectively has no sitemap of its own.

Type your address with /robots.txt after it. There is a line starting with Sitemap:. Does it name the same domain you are on right now?

What this has to do with AI

People with a question now type it into a language model about as often as into Google. A model like that fetches pages, reads what it can read, and sometimes names the source. To be named you need three things: your page has to exist, it has to be readable without JavaScript, and it has to be clear that it is you making the claim.

That last one is what structured data exists for. A block in the page source that says: this is an article, this is the title, this is the date, and this company is the author. That is not a trick. It is the only way a machine can tell your explanation apart from the menu next to it.

Frequently asked questions

How do I know whether my articles have their own address?

Open your news page and click a post. If the address in the bar changes to one that belongs to that post alone, you are fine. If you get a PDF, or you stay on the same page, you are not.

So a PDF is worthless?

No. Google indexes PDFs, and for a manual or a spec sheet it is a fine format. As an article it is a poor one: no title of its own in the results, no structured data, and harder to read for the crawlers that language models use.

Do I have to rebuild my whole site?

Usually not. What you need is one page type for articles, with its own address and a title. The rest of the site can stay as it is.

Am I better off blocking AI crawlers?

That is a choice, and there is no single right answer. Block them and you will not be named in the answers those models give. If you want to be found, blocking is the opposite of what you want. Look at your robots.txt to see what it says now, because often nobody knows.