
Google Analytics vs Google Search Console (2026)
Search Console shows what happens before someone reaches your site from Google. Analytics shows what happens after they arrive. Here is the full comparison.

Last updated: November 2026
Learning how to create a robots.txt file is a small job, and it gives you real control over which parts of your site search engines crawl. The file is a plain text file at the root of your website that tells search engine bots which pages they may visit and which ones they should skip. If you are not sure what this file does yet, our guide to what robots.txt is explains it in plain words.
In this guide, you will check whether your site already has one, write a simple file in a few minutes, upload it to the right place, and test it so you know it works. You do not need to know how to code.
On this page:
Before you create anything, check whether your site already has a robots.txt file. WordPress and most website platforms create one for you, so you may only need to edit it.
Open your browser and type your domain followed by /robots.txt, such as example.com/robots.txt. If you see a page of text with lines like User-agent and Disallow, your site already has a file. If you see a 404 error page, it does not have one yet.
A missing file is not a problem. When a site has no robots.txt file, crawlers may crawl everything they can reach. You only need a file when you want to change that, for example by keeping bots out of your admin pages.
If your site already has a file, read it before you change it, and save a copy on your computer first so you can put it back if something goes wrong.

A robots.txt file is a plain text file. It must be named exactly robots.txt, all lowercase, and saved in UTF-8 format. A file named Robots.txt, robots.TXT, or robots.txt.txt will not work, because crawlers only look for the exact name in lowercase.
Open a plain text editor. On Windows, use Notepad. On a Mac, use TextEdit, but switch it to plain text mode first (Format, then Make Plain Text). Never use Microsoft Word or Google Docs for this. Word processors add hidden formatting that breaks the file.
Start with an empty file, type or paste your rules from the next step, then save it as robots.txt in plain text, UTF-8 format.
Keep the file small. Google reads roughly the first 500 KiB of a robots.txt file, and most sites need only a few lines.
A robots.txt file is made of groups of rules. Each group starts with a User-agent line, followed by Disallow or Allow lines. You can also add a Sitemap line.
Here is what the four parts do:
Here is a starter example for a normal website. It lets all bots crawl the site, keeps them out of the admin area, and points them to the sitemap:
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://example.com/sitemap.xml
Replace example.com with your own domain and the sitemap address with your real one. If you use an SEO plugin like Rank Math or Yoast, it creates a sitemap for you, and you can find its address in the plugin settings.
You can add more Disallow lines for pages that have no value in search results, such as internal search pages, shopping cart and checkout pages, and thank-you pages after a form is sent. Only block pages that you truly do not want in search results.
One warning: robots.txt is a request, not a lock. Good bots follow it, but it does not hide pages from people, and it does not, on its own, keep a page out of search results. If you need to keep a page private, use a password or a noindex tag.
For the full official rules, Google’s robots.txt guide explains each part with examples, and the formal standard is set out in RFC 9309.
If your site runs on WordPress, you usually do not need FTP or a file manager. Your SEO plugin can create and edit the file for you.
With Rank Math: go to your WordPress dashboard, then Rank Math, then General Settings, then Edit robots.txt. Edit the rules in the text box, then save.
With Yoast SEO: go to your WordPress dashboard, then Yoast SEO, then Tools, then File Editor. If you see a robots.txt tab, click it, edit the rules, and save.
If you would rather work with a real file, create robots.txt on your computer and upload it through your hosting file manager or FTP. Just be careful: if your SEO plugin also manages a virtual file, the plugin version may take priority over the uploaded file. Pick one method and stick to it.
After you save, open yourdomain.com/robots.txt in your browser and check that your changes show up. If you see the old version, clear your cache and check again.

If you created the file on your computer, you need to put it in the root folder of your website. The root folder is the main folder that holds your homepage files. In most hosting accounts, it is called public_html.
You can upload the file in two ways. The easy way is your hosting file manager: log in to your hosting control panel, open the file manager, go to the public_html folder, and upload robots.txt there. The other way is FTP: connect with a program like FileZilla and drag the file into the root folder.
After the upload, open yourdomain.com/robots.txt in your browser. You should see the exact text you wrote. If you see an old version or a 404 page, the file may be in the wrong folder. Fix the location, clear the cache, and check again.
Creating the file is only half the job. You also need to test it, because a small typo can block pages you want in search results.
Start with the simplest test: open yourdomain.com/robots.txt in your browser and read it line by line. Look for typos and missing slashes. A single Disallow: / line, with nothing after the slash, blocks your whole site, so check for that first.
Next, test it in Google Search Console. Go to Settings and open the robots.txt report. Google shows you the file it last fetched, the date it fetched it, and any errors it found. If the report shows an old version, request a recrawl.
Then spot-check a page with the URL Inspection tool. Paste the address of a page, run the test, and look at the crawl section to see whether robots.txt allows or blocks it. Test one page you want indexed and one you meant to block.
Finally, watch the results over the next few days. Check the Pages report in Search Console for any new “Blocked by robots.txt” warnings on pages that should be indexed. If you see one, open your file and fix the rule that caused it.
Most robots.txt problems come from a short list of simple errors.
The most damaging mistake is a rule that blocks everything. This happens with a line like Disallow: / with no path after the slash. Before you save, read every Disallow line and ask what it blocks. If you are not sure, test it with the URL Inspection tool in Search Console first.
Robots.txt does not make a page private. It only asks good bots not to crawl it. The page address is still public, anyone can open it, and it can still show up in search results if other sites link to it. To keep a page out of search results, use a noindex tag. To keep people out, protect it with a password or a login.
The file must be named robots.txt, all lowercase. Robots.txt, ROBOTS.TXT, and robots.txt.txt are all wrong, and crawlers will treat the site as if it has no file at all.
Crawlers only look for robots.txt at the root of your site. A file inside /blog/, /public/, or any other folder is ignored.
Google needs your CSS and JavaScript files to render your pages the way visitors see them. If you block those files, Google may judge your pages on a broken version. Do not block your theme files, your CSS, or your JavaScript unless you have a clear reason.
Each subdomain needs its own robots.txt file. Rules for example.com do not apply to blog.example.com or shop.example.com.
Many site owners now ask about AI crawlers, the bots that collect content to train AI models and power AI search answers. The two best-known ones are GPTBot, used by OpenAI, and ClaudeBot, used by Anthropic.
You can block these bots in the same file, with their own User-agent groups. For example, these lines tell GPTBot and ClaudeBot not to crawl your site:
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: /
Blocking AI crawlers only stops those bots from crawling. It does not remove content that was already collected, and it does not stop your pages from showing up in normal search results. Keeping them out protects your content from AI training, but letting them in can bring your site into AI answers. There is no single right choice, and you can change the rules later.
One more thing to note: AI crawlers do not all follow robots.txt the same way. The well-known ones do, but smaller bots may ignore it.
Not always. If you have no pages you want to keep out of crawling, your site works fine without one. You need a file when you want to block admin pages or other low-value pages, or point bots to your sitemap.
It goes in the root folder of your website, usually called public_html. It must be at yourdomain.com/robots.txt. A file in any subfolder will be ignored.
Google usually fetches a new robots.txt file within a day or two for small sites, but it can take longer on large sites. You can speed things up in Search Console by opening Settings, then the robots.txt report, and requesting a recrawl.
Not on its own. Robots.txt tells bots not to crawl a page, but a page can still show up in search results if other sites link to it. To remove a page from search results, add a noindex tag to the page itself.
Google ignores lines it cannot read and follows the rest, and it reports the error in the robots.txt report. But a serious error, such as a rule that blocks your whole site, can keep your pages out of search results until you fix it.
A good robots.txt file is short, clear, and tested. Spend ten minutes writing it and five minutes checking it, and you will have a file that helps search engines crawl the pages that matter and skip the ones that do not.

Search Console shows what happens before someone reaches your site from Google. Analytics shows what happens after they arrive. Here is the full comparison.

Find out who really outranks you on Google, what they do better, and where the gaps are — with free tools and a simple 9-step plan.

Learn how to improve website speed with 13 simple steps using free tools. Fix images, caching, plugins, and hosting — no coding needed.