What robots.txt can and cannot do

robots.txt is a request, not a lock. Well-behaved crawlers read it and obey; badly behaved ones ignore it entirely. It controls crawling, not indexing. A page blocked in robots.txt can still appear in search results if other sites link to it — Google simply shows it without a description, because it was never allowed to look.

If you genuinely need a page kept out of the index, use a noindex meta tag or an HTTP header and let crawlers reach the page to see it. Blocking the URL in robots.txt actively prevents the noindex tag from ever being read.

The rules worth knowing

Paths are matched from the start of the URL, so Disallow: /cart also blocks /cart-abandoned/. Use a trailing slash when you mean a directory. The asterisk matches any sequence of characters and the dollar sign anchors the end, so Disallow: /*.pdf$ blocks every PDF.

Where two rules conflict, Google applies the most specific one, not the first one. That is why an Allow for a single file inside a disallowed directory works exactly as you would hope.

Blank lines separate groups. A User-agent line starts a new group, and every rule under it belongs to that agent until the next one. Getting this wrong is the single most common robots.txt mistake — rules stranded under the wrong agent do nothing at all.

A sensible starting point for WordPress

Most WordPress sites need very little: allow everything, disallow /wp-admin/, explicitly allow /wp-admin/admin-ajax.php so front-end features keep working, and point to your sitemap. Blocking /wp-includes/ or your theme directory is an old habit that now does more harm than good, because it stops Google rendering your pages properly.