Meregate.com
Wednesday, November 18, 2026
Home › SEO › What Is Robots.txt? A Simple Guide for Beginners (2026)

What Is Robots.txt? A Simple Guide for Beginners (2026)

What Is Robots.txt? A Simple Guide for Beginners (2026)

If you run a website, you have probably heard of robots.txt. It is a small file, but it plays a big part in how search engines find and read your site. This guide explains what is robots.txt in plain words, how it works, and what it can and cannot do.

Think of your website as a shop. Search engines send out small programs called crawlers to visit your shop and look at every page, so the pages can show up in search results. A robots.txt file is a note you leave at the front door. It tells those crawlers which rooms they may enter and which rooms they should skip. It does not lock the doors. It is a polite request, and most honest crawlers follow it.

What Is Robots.txt?

Robots.txt is a plain text file that sits at the root of your website. It gives instructions to web crawlers, also called robots or spiders, about which parts of your site they are allowed to crawl. The file follows the Robots Exclusion Protocol, a convention from 1994 that became an official internet standard, RFC 9309, in 2022.

Nearly every website has one. You can check any site by adding /robots.txt to the end of its home page address. If the file does not exist, crawlers will simply crawl the site as normal. Nothing breaks. Each line in the file is a rule, and a basic file can be as short as two or three lines.

Where Does Robots.txt Live?

A robots.txt file must live at the root of your domain. That means it goes at the top level, not inside a folder. For your site, the address is always yourdomain.com/robots.txt. Crawlers know to look there first before they start crawling your pages. If you put the file anywhere else, crawlers will not find it. A file at yourdomain.com/blog/robots.txt does nothing.

Most content management systems, like WordPress, create a basic robots.txt file for you, and many SEO plugins let you edit it from your dashboard.

How Does Robots.txt Work?

Before a crawler visits any page on your site, it asks for your robots.txt file. It reads the rules, then crawls only the parts you allow. If a page is blocked in the file, a good crawler will skip it. If there is no file, the crawler treats your whole site as open. The file uses a few simple commands, one per line. Here are the ones you will see most often.

User-agent

The User-agent line says which crawler the rules apply to. Every crawler has a name. Google’s main crawler is called Googlebot. A star (*) means the rules apply to every crawler, and that is the most common setup for small sites. If you want rules only for Google, write User-agent: Googlebot instead.

Disallow

The Disallow line tells crawlers which pages or folders they should not crawl. This is the most used command in a robots.txt file. You list the path of the page or folder after the word Disallow. For example, Disallow: /wp-admin/ tells crawlers to stay out of the admin area of a WordPress site. Disallow: /checkout/ keeps them out of checkout pages on an online shop.

Allow

The Allow line does the opposite. It tells crawlers that a page or folder is fine to crawl, even if a wider Disallow rule covers it. This is useful when you block a whole folder but want to let crawlers into one page inside it. Not every crawler supports Allow, but Google and Bing do.

Sitemap

The Sitemap line points crawlers to your sitemap. A sitemap is a list of the pages on your site that you want search engines to find. It helps crawlers find new and updated pages faster. The line looks like this: Sitemap: https://yourdomain.com/sitemap.xml. You can list more than one sitemap if your site has several. This line can sit anywhere in the file.

Colorful lines of code on a computer screen
Source: Wikimedia Commons

A Simple Example Robots.txt File

Here is a short example for a small business website. It blocks the admin area and the internal search results, allows everything else, and points crawlers to the sitemap.

User-agent: *
Disallow: /wp-admin/
Disallow: /search/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourdomain.com/sitemap.xml

Let us read it line by line. The first line says the rules apply to all crawlers. The second line keeps crawlers out of the WordPress admin area. The third line keeps them out of internal search result pages. The fourth line makes one exception, so a file the site needs stays open. The last line points crawlers to the sitemap. Most small websites need only a file like this. A short, clear file is easier to keep correct, and a wrong rule in a long file can block pages you want in search results.

What Is Robots.txt Used For?

Robots.txt has a few practical jobs. The first is keeping crawlers out of pages that have no value in search results: admin pages, login pages, shopping cart pages, and checkout pages. Nobody searches Google for your login page, so there is no reason to let crawlers spend time on it.

The second job is keeping crawlers out of internal search results. When someone searches inside your own site, your site creates a search results page. If crawlers visit those pages, they can end up in Google’s index. Those pages are thin and unhelpful, so blocking them keeps your search results clean.

The third job is saving crawl time. Search engines do not crawl your whole site every day. If crawlers waste time on admin pages, duplicate pages, and filters, they may miss your new blog posts and product pages. A robots.txt file points them toward the pages that matter. It also points crawlers to your sitemap, which helps new pages get found faster.

The Big Misunderstanding: Crawling Is Not Indexing

This is the most important part of this guide. Robots.txt controls crawling, not indexing. Those are two different things, and mixing them up causes real problems.

Crawling is when a search engine visits a page and reads it. Indexing is when the search engine stores that page so it can show up in search results. A robots.txt file can stop the first step, but it does not stop the second step on its own.

If you block a page in robots.txt, Google will not crawl it. But Google may still show that page in search results. It can learn about the page from links on other sites. The result will usually show without a description, just the page address.

If you want to keep a page out of search results completely, robots.txt is the wrong tool. Use a noindex tag instead. A noindex tag is a small piece of code you add to the page itself. It tells search engines not to show the page in results. The page must be crawlable for the noindex tag to be seen, so the two tools work in opposite ways. Block a page in robots.txt, and crawlers never see the noindex tag on it.

The rule is simple. Use robots.txt to control which pages crawlers visit. Use noindex to control which pages show up in search results.

Close-up of HTML and JavaScript code on a screen
Source: Wikimedia Commons

Things to Know Before You Edit Your File

It is a request, not a lock

Robots.txt is based on trust. Good crawlers like Googlebot and Bingbot follow the rules. But a bad crawler can ignore the file completely. The file does not hide your pages or protect them with a password. If a page must stay private, protect it with a login or a password. The file is also public: anyone can read it at yourdomain.com/robots.txt, so never list private folders in it.

Google reads only the first 500 KiB

Google reads only the first 500 kilobytes of a robots.txt file. Anything after that is ignored. For most small sites this limit never matters, because the file is only a few lines long.

Google stopped supporting Noindex in robots.txt

In the past, some site owners used a Noindex line inside robots.txt to keep pages out of search results. Google stopped supporting that in September 2019. If your file still has Noindex lines, remove them and use a noindex tag on the page itself instead.

AI crawlers read it too

Search engine crawlers are not the only ones that read robots.txt. AI companies send crawlers to collect content for training and for AI search tools. OpenAI’s GPTBot is one example. These crawlers also read robots.txt and, in most cases, follow it. To block an AI crawler, add a User-agent rule for it, just like you would for Googlebot.

How to Check and Create Your Robots.txt File

Checking your file takes less than a minute. Open your browser and type your site address followed by /robots.txt. If a file exists, you will see its text. If you see a 404 error, your site has no robots.txt file yet.

To create or edit the file, use your SEO plugin if you run WordPress. Most SEO plugins, including Rank Math, have a robots.txt editor in your dashboard. Or create a plain text file named robots.txt and upload it to the root folder of your site.

After you make a change, test it. Google Search Console has a free robots.txt tester in its settings. Check that the pages you want blocked are blocked and the pages you want open are open. Test before you save a big change, because one wrong line can block your whole site from search engines.

For the full technical details, see Google’s own guide to robots.txt.

Frequently Asked Questions

What is robots.txt in simple words?

Robots.txt is a small text file on your website that tells search engine crawlers which pages they may visit and which they should skip. It sits at yourdomain.com/robots.txt and uses simple rules like User-agent and Disallow.

Do I need a robots.txt file?

Most websites benefit from having one, but your site will still work without it. If you have no file, crawlers will crawl your whole site. A basic file keeps crawlers out of admin pages and internal search results, and points them to your sitemap.

Does robots.txt hide a page from Google search results?

No. Robots.txt only stops crawlers from visiting a page. Google can still show a blocked page in search results if it learns about it from links elsewhere. To keep a page out of search results, use a noindex tag on the page instead.

Can I block AI crawlers with robots.txt?

Yes. AI crawlers such as GPTBot read robots.txt like other crawlers do. You can block one by adding a User-agent rule with its name and a Disallow line for the whole site. Most major AI crawlers respect the file, though a bad one can ignore it.

Taylor Orlando

Taylor Orlando is a content writer at Meregate covering websites, SEO, and AI. Taylor breaks down complex topics into simple, practical guides that help small businesses build better websites, rank higher on Google, and get real value from AI tools — no jargon, no fluff.

Related Articles