WordPress Robots.txt: What to Block (and What Never to Touch)

A single wrong line in your WordPress robots.txt file can quietly remove your homepage from Google, or let a crawler burn through your server resources scanning pages nobody needs indexed. WordPress robots.txt best practices come down to a short list of rules: what to block, what to leave alone, and now, in 2026, whether to block AI crawlers at all. Get it wrong in either direction and you either waste crawl budget on junk pages or accidentally deindex the pages that pay the bills.

What Does a Robots.txt File Actually Do on WordPress?

A robots.txt file is a plain text file at the root of a website that tells search engine crawlers and other bots which parts of the site they’re allowed to request. It does not remove pages from Google’s index by itself, and it cannot force a bot to obey it, since well-behaved crawlers like Googlebot respect it while malicious bots typically ignore it entirely.

Every WordPress site has a virtual robots.txt file generated automatically, even if nobody ever created one. It’s viewable at yourdomain.com/robots.txt, and most SEO plugins, including Rank Math and Yoast, let you edit it directly from the WordPress dashboard instead of uploading a file over FTP.

What Should You Block in a WordPress Robots.txt File?

The pages worth blocking in a WordPress robots.txt file are the ones that offer nothing to a searcher and just waste crawl budget: admin URLs, internal search results, and cart or checkout pages on ecommerce sites. None of these pages are meant to rank, so letting bots crawl them repeatedly pulls attention away from the pages that actually matter.

• /wp-admin/ — the entire admin area, with one narrow exception below
• /?s= or /search/ — internal search result pages, which create thin, duplicate content with zero unique value
• /cart/, /checkout/, /my-account/ — WooCommerce utility pages that should never show up in search results
• /wp-includes/ and /wp-content/plugins/ — core and plugin files with no content value for search

Blocking a page in robots.txt is not the same as removing it from Google’s index. If a blocked page was already indexed, it can still appear in search results with no description, because Google respects the crawl block but may still know the URL exists from other links. Deindexing a page properly requires a noindex meta tag or password protection instead, which is a distinction that trips up a lot of site owners trying to hide a page from search.

What You Should Never Block in Robots.txt

Never block /wp-content/uploads/, your theme’s CSS and JS folders, or /wp-admin/admin-ajax.php, because blocking any of these breaks how Google renders your pages. Google needs to load your stylesheets, scripts, and images to evaluate Core Web Vitals and mobile-friendliness, and a page that renders as unstyled text in Google’s eyes gets treated as a worse page than it actually is.

A common and costly mistake is copying a robots.txt template from a tutorial written for a different platform or an old WordPress version, then blocking folders that no longer exist or that WordPress now depends on for front-end functionality. The admin-ajax.php file is the clearest example: it lives inside /wp-admin/, but plenty of themes and plugins rely on it to load content dynamically on the front end, so it needs an explicit allow rule even while the rest of the admin folder stays blocked.

Also avoid blocking entire blog or product directories by accident. It happens more often than it should when a site owner disallows a folder to hide one page and doesn’t realize the rule matches every URL underneath it, silently pulling dozens of published posts or products out of consideration for ranking.

Should You Block AI Crawlers Like GPTBot in Robots.txt?

Whether to block AI crawlers depends on what you’re trying to protect: block them if you don’t want your content used to train AI models, but understand that blocking the wrong bot can also cut you out of being cited by AI answer engines like ChatGPT and Perplexity. These are not the same category of bot, even though they’re often lumped together as “AI crawlers.”

According to Elementor’s robots.txt guide, major AI companies including OpenAI, Anthropic, and Google respect robots.txt directives for their training crawlers, which means a Disallow rule aimed at a bot like GPTBot generally works as intended. Training crawlers and retrieval crawlers serve different purposes, though, and treating them identically in your robots.txt file can backfire.

For a portfolio, agency, or small business site that wants visibility wherever potential clients are searching, including inside AI chat tools, blocking retrieval-focused bots like OAI-SearchBot or PerplexityBot usually works against the goal. Blocking pure training crawlers like GPTBot is a separate, more defensible call, since it doesn’t affect whether AI answer engines can cite a page right now.

How to Edit Your WordPress Robots.txt File

The safest way to edit a WordPress robots.txt file is through an SEO plugin’s built-in editor rather than uploading a file manually, since it avoids conflicts with the virtual file WordPress generates by default. Both Rank Math and Yoast SEO include this feature, and the steps are nearly identical between them.

1. Open the SEO plugin’s settings, usually under Rank Math > General Settings or Yoast SEO > Tools, and look for “Edit robots.txt”
2. If a physical robots.txt file doesn’t already exist in the site’s root folder, the plugin creates one when the edit is saved
3. Add Disallow rules for the folders that shouldn’t be crawled, one per line
4. Add an explicit Allow: /wp-admin/admin-ajax.php line if the rest of /wp-admin/ is blocked
5. Include a line pointing to the XML sitemap so crawlers can find it without guessing
6. Save, then check the file directly at yourdomain.com/robots.txt to confirm the changes are live

After editing robots.txt, checking how Google is actually crawling and indexing the site is worth doing in Google Search Console, since the Page Indexing report will flag any pages Google can’t reach because of a crawl block.

A Sample WordPress Robots.txt File

This is a reasonable starting point for a standard WordPress site without WooCommerce. Adjust the ecommerce-specific lines out if they don’t apply, and always swap in the site’s real sitemap URL.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Disallow: /wp-content/plugins/
Disallow: /wp-includes/

Sitemap: https://yourdomain.com/sitemap_index.xml

Notice what’s absent from this file: no blanket block on /wp-content/, no block on CSS or JS folders, and no attempt to hide entire post or product categories. A robots.txt file that tries to do too much usually ends up doing damage instead of protection, and the fix from there involves getting the site’s URL structure and crawl rules working together instead of against each other.

Robots.txt is one small file, but it sits upstream of everything else an SEO plugin like Rank Math or Yoast tries to do. Get the crawl rules wrong and no amount of meta title optimization or schema markup fixes the fact that Google can’t reach the page in the first place.

Frequently Asked Questions