One wrong line in this file will remove your entire site from Google.
robots.txt is a text file in the root directory of your website that tells search engines which parts they can and cannot visit. It's an instruction that serious crawlers respect, not security. One wrong rule can keep your entire site out of search results.
IN COMMON LANGUAGE
What it does and doesn't do
It controls access to crawling, not viewing. Anyone can read the file, so never put paths in it that you want to keep secret: you will then publish exactly where they are.
It is also not a way to keep pages out of the index. You use a noindex instruction for this; a blocked page may still appear in the results without a description.
WHY IT MATTERS
Where things go wrong
The classic error is a blockage that was part of the test environment and remains after go-live. Your site will then gradually disappear from Google without an error message.
Therefore, check after each go-live whether robots.txt allows what it should allow. It takes ten seconds.
- Check after go-live — the most expensive forgotten institution.
- No mention of secret paths — the file is public.
- Do not use to de-index — that's what noindex is for.
- Refer to your sitemap — so crawlers find it.
- Don't block CSS or scripts — then Google cannot rate your page.
FREQUENTLY ASKED QUESTIONS
More about robots.txt
How do I check my robots.txt?
Type your domain followed by /robots.txt in your browser. If you see 'Disallow: /' under 'User agent: *', you are blocking your entire site and that is almost certainly a mistake.
Should I block AI crawlers?
That's a choice. Blocking protects your texts against reuse and excludes you from being mentioned in AI answers. We consciously allow them because being mentioned benefits us more.
Do I need this file?
Not strictly, but it's wise: even if just to reference your sitemap. Without a file, anything can be crawled, and that is usually exactly the intention.
CONTINUE READING
Of course, continue reading about this subject.
Technically okay?
We check robots.txt, sitemap and indexation and send you what we find.