A robots.txt file is a plain text file placed at the root of your website that tells search engine crawlers which pages or sections they’re allowed to visit and which to skip. It’s not a security tool, but a set of polite instructions that well-behaved bots follow.
What Robots.txt Looks Like
A basic example uses “User-agent: *” to target all bots, followed by “Disallow: /wp-admin/” to block the admin area. You can also specify specific bots by name, like Googlebot.
Common WordPress Entries
- Block /wp-admin/ to keep the admin area out of search results
- Allow /wp-admin/admin-ajax.php so front-end AJAX features still work
- Block /wp-login.php to reduce spam bot traffic
Where to Find It
Visit yourdomain.com/robots.txt to see your current file. Yoast SEO and Rank Math let you edit your robots.txt from within the WordPress dashboard without using an FTP client.
Important Warnings
- A mistake in robots.txt can accidentally block Google from crawling your entire site
- Disallowing a URL doesn’t remove it from the index if other sites link to it
- Use the noindex meta tag for pages you want controlled in results
Related: XML Sitemap
Related: Technical SEO
