robots.txt is a request, not a lock. Well-behaved crawlers read it and comply; badly behaved ones ignore it entirely, and nothing about the file prevents anyone from fetching anything. Treating it as a security measure is the first mistake people make with it.
The second is more expensive: assuming it keeps pages out of search results. It does not, and the reason why is worth understanding before you write a single rule. The Robots.txt Generator will produce a valid file; this is how to decide what should go in it.
Crawling and indexing are different things
robots.txt controls whether a crawler fetches a URL. It has no direct say in whether that URL appears in search results. Those are separate systems, and the gap between them produces the most common robots.txt failure.
If you block a URL and another site links to it, Google can still list it — typically with no description, because it was never permitted to look at the page. The URL sits in the index, unhelpfully, and blocking it harder does not remove it.
Worse, blocking a page prevents Google from ever seeing a noindex tag you added to it. The instruction you wrote to remove the page lives inside a document the crawler is not allowed to open. Pages get stuck this way for months.
The rule that avoids all of it: to stop a page being crawled, use robots.txt. To stop it being indexed, allow the crawl and use noindex. Only block the URL afterwards, once it has actually dropped out.
The syntax, and the mistake in it
A robots.txt file is a series of groups. Each group starts with one or more User-agent lines and is followed by the rules that apply to those agents. A blank line ends the group.
User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /?s=
Sitemap: https://example.com/sitemap.xmlThe mistake is stranding rules under the wrong agent. If you add a User-agent: Googlebot group near the top and then write general rules beneath it, those rules apply to Googlebot alone — every other crawler falls through to the wildcard group and follows whatever is there instead.
Crawlers also obey exactly one group: the most specific one matching their name. Googlebot reading a file with both a Googlebot group and a wildcard group follows the Googlebot group and ignores the other completely. Rules are not inherited between groups, so anything Googlebot must obey has to be repeated inside its own group.
Path matching and precedence
Paths match from the beginning of the URL, which catches people constantly. Disallow: /cart also blocks /cart-abandoned/ and /cartography/. Add the trailing slash when you mean a directory.
- An asterisk matches any sequence of characters, so a rule ending in a question mark wildcard blocks every URL carrying a query string.
- A dollar sign anchors the end of the pattern, so a PDF rule written that way blocks
/file.pdfbut not/file.pdf?download=1. - An empty
Disallow:line means allow everything, and is the conventional way to write a permissive group. - Rules are case-sensitive on the path, matching how URLs behave on most servers.
Where two rules conflict, Google applies the most specific one, measured by the length of the path pattern, not the order they appear in the file. This is why an Allow for a single file inside a disallowed directory works exactly as you would hope, and why rule order matters far less than people assume.
If a path contains characters that need escaping, get that right before it goes in the file — the URL Encoder and Decoder will show you the correct form.
A sensible WordPress starting point
Most WordPress sites need remarkably little. Allow everything, block the admin area, explicitly allow admin-ajax.php so front-end features keep working, block internal search, and point at your sitemap.
User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /?s=
Sitemap: https://example.com/sitemap.xmlWhat not to do: blocking /wp-includes/, your theme directory, or your uploads folder. That advice circulated widely years ago and is now actively harmful, because Google renders pages to evaluate them and cannot do so without your CSS, JavaScript and images.
The Sitemap line is independent of user-agent groups, so its position in the file does not matter and you can list several. It is the cheapest way to make a sitemap discoverable without submitting it anywhere — the rest of that workflow is in Sitemap Generator: How to Create and Submit an XML Sitemap.
What blocking actually saves you
On a small site, nothing measurable. Google will crawl a few hundred pages regardless, and time spent tuning robots.txt is time not spent on the content.
On a large site, or one that generates URLs automatically, it is one of the few levers that works immediately. Faceted navigation, session parameters and internal search can multiply a 3,000-page site into hundreds of thousands of addresses, and blocking those patterns redirects crawler attention to pages that matter. That whole problem is the subject of How to Stop Crawl Waste From Eating Your Rankings.
Server load is the other real benefit. Aggressive crawling of infinite URL spaces costs you response time, and slow responses cause Google to lower its own crawl rate — a feedback loop worth breaking. If you are not sure how heavy those pages are to begin with, the Page Size Checker will tell you.
Test before you deploy
A robots.txt mistake is unusually costly because it is silent. Nothing errors, nothing warns you; traffic simply stops arriving over the following weeks. A blanket disallow left in place after a staging launch is the single most expensive typo in SEO.
- Generate the file and read it start to finish. Check every rule sits under the agent you intended.
- Test specific URLs against it in the Search Console robots.txt tester before uploading.
- Upload to the domain root so it answers at
/robots.txt. Subfolders are ignored, and every subdomain needs its own file. - Fetch the live file in a browser and confirm what you see is what you wrote.
- Check Search Console crawl stats a fortnight later to confirm the change did what you expected.