Learning how to create a robots.txt file takes about ten minutes, and most of that time is spent deciding what to block rather than writing the syntax. A robots.txt file is a plain text file at the root of your domain that tells search engine crawlers which URLs they may request. Get it right and you steer crawl budget toward pages that matter. Get one slash wrong and you can accidentally hide your entire site from Google.
This guide walks the whole job end to end: the four directives you actually need, the longest-match-wins rule that trips up almost everyone, real allow and disallow examples, exactly where to upload the file, and how to test it before a crawler ever sees it. No vague boilerplate, no copying Google's docs word for word.
What a Robots.txt File Actually Does#
A robots.txt file is a set of instructions for web crawlers, served from https://yourdomain.com/robots.txt. When a well-behaved bot like Googlebot visits your site, it reads this file first and obeys the rules it finds before requesting any other URL.
The key word is "well-behaved." Robots.txt is a request, not a wall. Google, Bing, and other major engines honor it. Scrapers and malicious bots can ignore it entirely. So treat robots.txt as crawl guidance, never as a security control.
Important: robots.txt controls crawling, not indexing. Blocking a URL here does not guarantee it stays out of search results. If Google finds links to a blocked page elsewhere, it can still list the URL (without a description). To keep a page out of results, you need a noindex tag, which is a separate mechanism we cover near the end.
The four parts you actually use#
Most real-world files are built from just four directives:
- User-agent: which crawler the following rules apply to.
*means all crawlers. - Disallow: a URL path the crawler should not request.
- Allow: a URL path the crawler may request, used to carve exceptions out of a broader Disallow.
- Sitemap: the full URL of your XML sitemap, so crawlers find all your pages faster.
There are a couple of extras (Crawl-delay, host directives) that we will touch on, but those four cover 95% of sites.
How to Create a Robots.txt File Step by Step#
Here is the complete procedure, from a blank file to a tested, live robots.txt. Follow it in order and you will not paint yourself into a corner.
Step 1: Map what should and should not be crawled#
Before typing a single line, list your URL patterns. Open your site structure and sort pages into two buckets.
Things you usually want crawled: your homepage, blog posts, product or service pages, category pages, and your sitemap. Things you often want to keep out of crawl: admin areas (/admin/, /wp-admin/), internal search results (/search?), cart and checkout pages, thank-you pages, staging or duplicate URLs, and faceted-filter URLs that create thousands of near-duplicate combinations.
This map is the real work. The syntax is trivial once you know your intent.
Step 2: Write the user-agent block#
Start with the broadest rule that applies to every crawler:
User-agent: *
Every directive after this line, until the next User-agent line or end of file, applies to all crawlers. If you need a rule for one specific bot, add a separate block. For example, to give Googlebot different treatment from everyone else:
User-agent: Googlebot
Disallow: /private/
User-agent: *
Disallow: /private/
Disallow: /tmp/
A crawler reads only the most specific block that matches its name. Googlebot here obeys the first block and ignores the second entirely. That surprises people, so remember: a bot follows one group, not the combination.
Step 3: Add your Disallow and Allow rules#
Now translate your map from Step 1 into directives. Each Disallow takes a URL path that starts with /. A few patterns you will use constantly:
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /search
Allow: /search/help
Disallow: /*.pdf$
Reading that line by line:
Disallow: /admin/blocks everything under the admin folder.Disallow: /searchblocks any path beginning with/search, including/search?q=shoes.Allow: /search/helpcarves out one exception so that page stays crawlable.Disallow: /*.pdf$uses a wildcard (*) and an end-of-URL anchor ($) to block every PDF.
An empty Disallow: (nothing after the colon) means "block nothing," which is how you explicitly allow full crawling.
Step 4: Add the sitemap line#
Tell crawlers where your sitemap lives. Use the absolute URL, not a relative path, and put it anywhere in the file (convention is the bottom):
Sitemap: https://yourdomain.com/sitemap.xml
You can list more than one sitemap on separate Sitemap: lines. If you have not built one yet, generate it with the free XML sitemap generator and drop the resulting URL here. The sitemap line works regardless of which User-agent blocks exist, because it is global.
Step 5: Generate, save, and validate the file#
You can hand-write the file in any plain text editor (save it as UTF-8, named exactly robots.txt, with no .txt.txt double extension). If you would rather not risk a typo, the free robots.txt generator builds a syntactically valid file from your allow and disallow choices and previews how Googlebot reads each rule. That preview step is where you catch the precedence mistakes covered in the next section.
Step 6: Upload it to your domain root#
The file must be reachable at exactly https://yourdomain.com/robots.txt. It only works at the root of a host. A file at https://yourdomain.com/blog/robots.txt is ignored.
How you upload depends on your stack:
- WordPress: use an SEO plugin (Yoast, Rank Math) that exposes a robots.txt editor, or upload via FTP to the root directory.
- Static or custom site: place
robots.txtin your public or web root folder (oftenpublic/,www/, orhtdocs/). - Subdomains: each subdomain needs its own file.
blog.yourdomain.comandyourdomain.comdo not share one.
After uploading, visit the URL in your browser. You should see your plain text, not a 404 or your homepage.
Step 7: Test before you trust it#
Open Google Search Console and use the robots.txt report (it shows the file Google last fetched and flags syntax errors). Then test specific URLs: paste a path you meant to block and a path you meant to keep, and confirm each returns the result you expect. Testing one blocked URL and one allowed URL catches the vast majority of mistakes before they cost you traffic.
The Longest-Match-Wins Rule (Where Sites Break)#
This single rule causes more accidental blocks than anything else, and most tutorials skip it. When multiple Allow and Disallow rules could apply to the same URL, Google does not read top to bottom. It applies the rule with the longest matching path. If two rules tie on length, Allow wins.
Picture this file:
User-agent: *
Disallow: /blog/
Allow: /blog/public/
For the URL /blog/public/post-1, two rules match. Disallow: /blog/ matches 6 characters; Allow: /blog/public/ matches 13 characters. The longer rule wins, so this page is crawlable, even though a top-to-bottom reading might suggest it is blocked.
Now flip it and watch the trap spring:
User-agent: *
Allow: /products/
Disallow: /products/secret/
For /products/secret/item, the Disallow path is longer, so that URL is correctly blocked while the rest of /products/ stays open. The rules are doing exactly what the lengths dictate.
Here is the comparison that matters when you debug a "why is this page blocked" mystery:
| URL requested | Matching rules | Winner | Result |
|---|---|---|---|
/blog/public/post | Disallow: /blog/ (6), Allow: /blog/public/ (13) | Allow (longer) | Crawled |
/products/secret/x | Allow: /products/ (10), Disallow: /products/secret/ (18) | Disallow (longer) | Blocked |
/page | Allow: /page (5), Disallow: /page (5) | Allow (tie goes to Allow) | Crawled |
Warning: the most damaging mistake is a stray
Disallow: /sitting underUser-agent: *. A single slash blocks your entire site from every crawler. If your organic traffic ever falls off a cliff after a deploy, check robots.txt for that line first.
Real Robots.txt Examples You Can Adapt#
Concrete templates beat abstract rules. Here are three honest starting points. Swap the domain and adjust the paths to your structure.
A simple site that wants everything crawled#
User-agent: *
Disallow:
Sitemap: https://yourdomain.com/sitemap.xml
The empty Disallow: allows all crawling. This is the right default for a small blog or brochure site. Do not over-engineer it.
A WordPress or CMS site#
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Sitemap: https://yourdomain.com/sitemap_index.xml
The Allow line for admin-ajax.php matters: many themes and plugins need that endpoint, and blocking the whole /wp-admin/ folder without it can break front-end functionality crawlers rely on.
An e-commerce site managing crawl budget#
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /*&color=
Allow: /products/
Sitemap: https://yourdomain.com/sitemap.xml
The wildcard rules block faceted-filter URLs (sort, filter, color combinations) that otherwise spawn endless near-duplicate pages and waste crawl budget on large catalogs. The Allow: /products/ keeps your clean product URLs open.
Should you use Crawl-delay?#
Crawl-delay asks a crawler to wait a set number of seconds between requests. Google ignores it entirely (control Googlebot's crawl rate in Search Console instead). Bing and Yandex honor it. Only add it if a specific bot is hammering a small server, and even then, prefer fixing the server. A line like Crawl-delay: 10 under a specific user-agent is the rare case where it is worth it.
Crawling vs Indexing: The Distinction That Saves You#
The biggest conceptual error is treating robots.txt as a way to remove a page from Google. It is not. Disallowing a URL stops Google from crawling it, but a disallowed page can still appear in search results if other pages link to it, shown as a bare URL with no snippet.
Worse, there is a combination you must never use. If you add a noindex meta tag to a page and also block that page in robots.txt, Google can never crawl the page to see the noindex tag. The result is the page stays indexed indefinitely, the opposite of what you wanted.
The rule of thumb:
- To keep a page out of search results: use a
noindexmeta robots tag (orX-Robots-TagHTTP header for PDFs and other non-HTML files), and leave the page crawlable so Google can read that tag. - To keep crawlers off a section entirely (and accept it may still surface as a bare URL): use robots.txt
Disallow. - For sensitive content: do not rely on either. Use authentication.
If you are deciding which mechanism fits a given page, our explainer on blocking AI crawlers with robots.txt walks through the same crawl-versus-index logic for the new wave of AI training bots, which is an increasingly common reason people edit this file in 2026.
Common Mistakes and How to Avoid Them#
A short checklist of the errors that cost real traffic:
- A leftover
Disallow: /from staging. Development sites often block everything. Forgetting to remove it after launch nukes your visibility. - Blocking CSS and JavaScript. Google renders pages to understand them. Disallowing
/assets/or/js/can break rendering and hurt rankings. Let crawlers fetch your resources. - Wrong file location. It must be at the host root, lowercase
robots.txt, served astext/plain. - Relying on it for privacy. Listing
/secret-admin/in a public file is a roadmap for anyone curious. Use real access control. - Case sensitivity. Paths are case-sensitive.
/Blog/and/blog/are different rules.
When in doubt, change one rule at a time and re-test in Search Console. Slow and verified beats fast and broken.
Wrapping Up#
Now you know how to create a robots.txt file that does exactly what you intend: a clean set of user-agent, disallow, allow, and sitemap directives, placed at your domain root, tested before it goes live. The two ideas worth keeping in your head are the longest-match-wins precedence rule and the hard line between crawling (robots.txt) and indexing (noindex). Master those and you will never accidentally hide a site again.
When you are ready to build the file without hand-typing syntax, the robots.txt generator turns your allow and disallow choices into a valid file and previews how Googlebot reads it. Pair it with the XML sitemap generator so the Sitemap: line points somewhere real, and you have the full crawl-control setup in minutes.
Frequently Asked Questions#
Where do I put the robots.txt file?
At the root of your domain, reachable at exactly https://yourdomain.com/robots.txt. It must be lowercase, named robots.txt, and served as plain text. A file in a subfolder is ignored, and each subdomain needs its own separate file.
Do I even need a robots.txt file? Not strictly. If you are happy for search engines to crawl everything, you can skip it and crawlers will assume full access. Most sites still add one to point to their sitemap and to keep crawlers out of admin, cart, search, and duplicate URLs that waste crawl budget.
Will blocking a page in robots.txt remove it from Google? No. Robots.txt stops crawling, not indexing. A blocked page can still appear in results as a bare URL if other pages link to it. To remove a page from search results, use a noindex meta tag and keep the page crawlable so Google can read that tag. Never block a noindexed page in robots.txt, because then Google cannot see the noindex.
What does the asterisk in User-agent mean?
User-agent: * applies the rules that follow to all crawlers. If you add a block for a specific bot like User-agent: Googlebot, that bot follows only its own block and ignores the wildcard one. A crawler obeys a single group, not a merge of all matching groups.
How do I test my robots.txt before publishing? Use the robots.txt report in Google Search Console, which shows the file Google fetched and flags syntax errors. Test specific URLs by checking one path you meant to block and one you meant to keep crawlable. You can also preview rule behavior in a robots.txt generator before you upload, which catches longest-match precedence mistakes early.
Can I use wildcards in robots.txt?
Yes, within limits. * matches any sequence of characters, and $ anchors a rule to the end of a URL. So Disallow: /*.pdf$ blocks every PDF, and Disallow: /*?sort= blocks any URL containing a sort parameter. These two symbols are the only pattern matching the standard supports.
