Skip to content
Molixa

Robots.txt Generator

15 bots, AI crawler blocking, built-in path tester.

Share

Rules (1)

robots.txt

User-agent: *
Allow: /
Disallow: /admin
Disallow: /api
Disallow: /dashboard

Sitemap: https://example.com/sitemap.xml

Test: would this bot fetch this path?

BLOCKED -Path matches '/admin' which is disallowed for *. matched: Disallow: /admin
Domain, paths, business rules never leave your browser.
15
Bot presets
Yes
Built-in tester
Unlimited
Free uses per day
Browser
Runs in

What is the Robots.txt Generator?

A robots.txt file tells search engine crawlers what they can and can't fetch from your site. This generator builds a valid robots.txt from a form, with 15 common bots as presets (Googlebot, Bingbot, GPTBot, ClaudeBot for AI training, etc.) and 7 common disallow patterns. Then it tests your rules: paste a path, see whether Googlebot would fetch it, with the matched rule explained.

Most online generators give you a form and a download. We add a tester that implements robots.txt's longest-match-wins semantics per Google's official spec -so you can verify your rules before deploying.

How it works

Step 1
Pick bots
Choose from 15 presets or type a custom UA. Add Allow/Disallow paths per bot.
Step 2
Validate
Catches blocking-everything mistakes and missing sitemap. Errors before download.
Step 3
Test
Paste a path, choose a UA, see what would happen. Catches edge cases pre-deploy.

Features

15 bot presets
Googlebot, Bingbot, GPTBot, ClaudeBot, Google-Extended, CCBot, AhrefsBot, more.
7 disallow presets
Admin, API, search, cart, tracking params, tag pages -common with one click.
Path tester
Implements Google's longest-match-wins rule. Test before deploy.
Validation
Catches Disallow: / mistakes that would deindex you.
Sitemap directive
Multiple sitemaps supported. Absolute URLs validated.
Crawl-delay
Per-bot setting. Warning when set unreasonably high.
AI crawler blocking
Block GPTBot / ClaudeBot / CCBot to opt content out of AI training.
Browser only
Your domain, paths, business rules never leave the page.

Why this generator

Built-in tester

Most generators only emit a file. We let you verify it does what you intended before pushing to prod.

AI crawler awareness

GPTBot, ClaudeBot, Google-Extended are first-class presets -important in 2026 when AI training scraping is widespread.

Free, no ads

SEOptimer gates beyond their default at $29/mo. We ship the full feature set free.

Browser-only

Your domain map and disallow paths are competitive intel. We never see them.

Who uses it

SEO pros
Building robots.txt for client sites with AI-bot blocking.
Site owners
Want admin pages hidden from search, AI training opted out.
Web devs
Validating robots rules locally before deploying to a CDN.
Learners
Seeing what robots.txt syntax actually looks like, with a live tester.

Real use cases

  • Your /admin pages showed up in Google. Add Disallow: /admin under User-agent: *. Test that /admin/login is blocked, deploy.
  • You want to keep your blog out of OpenAI's training data. Add User-agent: GPTBot with Disallow: /. Test, deploy.
  • Your Bing crawl rate is killing your server. Add Crawl-delay: 5 under User-agent: Bingbot.
  • You have /admin disallowed but /admin/help is a public page. Add Allow: /admin/help. Tester confirms it's allowed.
  • You're launching a staging site. Disallow: / under * keeps Google out until you're ready.
  • Audit a competitor's robots.txt. Paste it into the tester to see what they block and don't.

Frequently asked questions

Is the robots.txt generator free?▾

Yes. Unlimited use, no signup, no daily cap. Generation and validation run entirely in your browser. SEOptimer's free generator caps you at one preset and shows ads; we ship 15 bots and 7 disallow presets free.

What is robots.txt for?▾

robots.txt is a text file at the root of your domain that tells search engine crawlers which paths they should and shouldn't fetch. It doesn't enforce security (anyone can ignore it) but well-behaved bots like Googlebot respect it.

Which bots can I configure?▾

15 common bots: all crawlers (*), Googlebot (and its sub-bots: Image, News), Bingbot, Yahoo Slurp, DuckDuckBot, Baiduspider, YandexBot, GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (AI training), CCBot (Common Crawl), AhrefsBot, SemrushBot. Block AI crawlers if you don't want your content used for training.

Does the tester actually simulate Googlebot?▾

It implements robots.txt's longest-match-wins rule per Google's official spec. If you have 'Allow: /admin/public' and 'Disallow: /admin', then /admin/public/file is allowed (longer pattern wins). The tester walks the same logic.

What's the difference between Allow and Disallow?▾

Disallow blocks paths; Allow explicitly permits them (used to whitelist sub-paths inside a disallowed parent). Most files only need Disallow rules. Allow is for the 'block /admin, but allow /admin/public-page' pattern.

Should I set Crawl-delay?▾

Probably not. Googlebot ignores Crawl-delay entirely (Google manages its own crawl rate). Bingbot, Yandex, Baidu accept it. Set 10s only if your server is being hammered. >30s slows indexing on bots that honor it.

Should I include the Sitemap directive?▾

Yes. Adding 'Sitemap: https://your-domain.com/sitemap.xml' helps search engines discover all your URLs. The URL must be absolute (full https://...). You can list multiple sitemaps if you split by section.

Can I block AI training crawlers?▾

Yes. Add User-agent rules for GPTBot, ClaudeBot, Google-Extended, CCBot with Disallow: / to opt your content out of AI training corpora. Note that not all AI scrapers respect robots.txt.

Where do I put robots.txt?▾

At the root of your domain: https://your-domain.com/robots.txt (not in a subfolder). Crawlers only check the root. If your site is on www.example.com, robots.txt goes at https://www.example.com/robots.txt.

Is my data sent anywhere?▾

No. The generator and tester run in your browser. Your domain, paths, and rules stay on your laptop. The downloaded robots.txt is yours alone -we don't see what you configure.

Generate robots.txt now

15 bots, 7 presets, built-in tester. Free unlimited.

Open the robots.txt generator

Related SEO tools

XML Sitemap GeneratorSchema Markup GeneratorSERP PreviewMeta Tag Generator
Built and reviewed byMolixa AI, Molixa AI
Last updated:

The Robots.txt Generator page is built, reviewed, and maintained by the Molixa team. We use the tool we ship and update the docs when the behavior changes.

Related SEO Tools

Meta Tag Generator

8-platform preview, pixel-accurate truncation, URL auto-fill, 4 output formats.

Schema Markup Generator

15 schema types, three formats (JSON-LD / Microdata / RDFa), built-in Rich Results validator.

SERP Preview Tool

Pixel-accurate desktop + mobile preview with length analysis.

Keyword Density Checker

1/2/3-grams + stop-word filter + page compare.

XML Sitemap Generator

Image + hreflang + sitemap-index, per-URL control, browser-only.

Popular Tools

AI Content Detector

Check if text was written by AI with a sentence-by-sentence heatmap.

YouTube Video Summarizer

Turn any YouTube video into clear notes with chapters, quotes, chat, and flashcards.

PDF Summarizer

PDF summarizer with page citations, multi-doc compare, and domain templates.

AI Text Rewriter

Paraphrase in 10 modes with diff view, freeze words, and brand voice training.

AI Math Solver

Free math solver with step-by-step solutions, photo upload, and 4 learning modes.

Object Remover

Brush over any unwanted object and remove it cleanly with AI.

From the blog