Robots.txt Generator

How Do You Make a robots.txt File for Your Website?

A free online robots.txt generator builds the text file that tells search engine crawlers which parts of your site they may visit. Choose a default rule, list the paths you want crawlers to skip, add your sitemap address if you have one, and press the button. The file text appears in your browser, ready to copy into a plain text file named robots.txt.

What Does the Robots.txt Generator Do and Why Is It Useful?

A robots.txt file sits at the root of a site, for example at https://example.com/robots.txt. Well-behaved crawlers request it before crawling and read its rules. It is a simple format, but small slips are costly. A stray slash can block an entire site, and a typo in a directive name can make a rule useless.

This tool removes the guesswork. You pick between allowing everything and blocking everything, list folders or pages to disallow, and optionally add a sitemap URL. It writes the lines in the correct order with the correct spelling. That makes it handy for new sites, for staging copies you do not want crawled, and for anyone who only touches robots.txt once or twice a year.

How Does the Tool Work Under the Hood?

The generator creates exactly one group of rules that applies to all crawlers. It starts with User-agent: *, where the asterisk means every crawler. What follows depends on your choices:

• With the default rule set to allow everything, the group has an empty Disallow: line unless you list paths, in which case each path gets its own Disallow: line.
• With the default rule set to block everything, the group contains Disallow: /, which covers the whole site.
• Each non-empty line in the paths box becomes one Disallow: line. Blank lines are ignored.
• If you enter a sitemap URL, a final Sitemap: line is added.

Everything happens after you press the button. The output does not change while you type, and the tool never contacts your site or checks that the paths exist.

How Do You Use the Robots.txt Generator Step by Step?

1. Choose the default rule: allow everything or block everything.
2. In the paths box, type one path per line that you want crawlers to avoid, such as /admin/ or /cart/. Start each path with a slash.
3. Optionally enter the full sitemap URL, beginning with https://.
4. Press the Generate button.
5. Select the output and copy it into a text file named exactly robots.txt.
6. Upload that file to the root folder of your website so it loads at /robots.txt.

What Do Real Examples Look Like?

Example 1: allow everything with a sitemap

Default rule: allow everything. Paths: none. Sitemap: https://example.com/sitemap.xml. The result is:

User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xml

An empty Disallow value means nothing is blocked.

Example 2: hide admin and cart pages

Default rule: allow everything. Paths: /admin/ on the first line and /cart/ on the second. The result is:

User-agent: *
Disallow: /admin/
Disallow: /cart/

Example 3: block a whole staging site

Default rule: block everything. The result is:

User-agent: *
Disallow: /

This asks every compliant crawler to stay away from the entire site, which suits a test copy that should not appear in search.

Example 4: a search results folder and a private download area

Paths: /search/ and /downloads/private/, with the sitemap https://example.com/sitemap_index.xml. The result is:

User-agent: *
Disallow: /search/
Disallow: /downloads/private/
Sitemap: https://example.com/sitemap_index.xml

Example 5: reading a result before you upload it

Say you chose allow everything, entered three paths (/admin/, /cart/ and /checkout/) and no sitemap. The output has one User-agent line and exactly three Disallow lines, so a quick count tells you the generator picked up every path you typed. If the number of Disallow lines does not match the number of paths, look for a path that was typed on the same line as another one.

The sample lines above show the structure. Treat the exact line breaks as a guide, since the tool may order or space lines slightly differently while keeping the same rules.

What Are Good Tips for Robots.txt?

• Keep the file small. Only block areas that truly add nothing to search, such as internal search pages, carts or login screens.
• Always start a path with a slash and end folders with one, so /admin/ means the folder and everything inside it.
• Include a sitemap line so crawlers can find your list of pages without guessing.
• Test the final file by opening yourdomain.com/robots.txt in a browser after uploading it.
• Review the file after site redesigns, because new folders may need new rules and old rules may become pointless.

How Does Robots.txt Fit With Other Search Settings?

Robots.txt controls crawling, meaning whether a bot requests a page at all. Other settings control indexing, meaning whether a page may be listed in results. A robots meta tag on the page, or a canonical link, works at the page level and is read only if the crawler is allowed to fetch the page. For that reason, do not block a page in robots.txt if you are relying on a noindex tag inside it, because the crawler will never see the tag. Think of robots.txt as a coarse tool for folders and meta tags as a precise tool for individual pages.

What Common Mistakes Should You Avoid?

• Blocking everything by accident. A single Disallow: / on a live site asks crawlers to skip every page. Check your default rule before uploading.
• Treating robots.txt as security. The file is a polite request. It does not hide anything, and the file itself is public, so listing a secret folder actually advertises it. Protect private content with passwords or server settings.
• Expecting removal from search. Blocking crawling does not necessarily remove a page that is already known. A noindex tag, which you can create with the Meta Tag Generator, is a different tool.
• Using the wrong file name or place. It must be called robots.txt in lower case and sit at the root, not in a subfolder.
• Relative sitemap URLs. The sitemap line should be a full address.

Who Uses a Robots.txt Generator?

Site owners launching a first website use it to avoid syntax errors. Developers use it to keep staging servers out of search. Online store owners use it to steer crawlers away from cart and checkout paths. Bloggers use it to point crawlers at a sitemap. When you are tidying URLs at the same time, the Slug Generator helps you create clean page addresses.

Is the Robots.txt Generator Private?

Yes. The tool runs completely in your browser. The paths and sitemap address you type are processed on your device and are not uploaded or stored. Nothing is sent to a server, and nothing is saved when you close the page.

What Are the Limitations?

• It creates a single group for all crawlers. There are no separate rules for individual user agents.
• It only writes Disallow rules and an optional Sitemap line. It does not generate Allow lines or crawl delay values.
• It does not check whether your paths exist, whether your sitemap loads or whether the file is live on your site.
• The file is only a request. Compliant crawlers follow it, but it cannot force any visitor or bot to obey.
• Results appear after you press the button, with no live updating, no download and no history. You copy the text yourself.

Frequently Asked Questions

Where do I put the robots.txt file?

Upload it to the top level of your domain so it opens at https://yourdomain.com/robots.txt.

Does robots.txt hide a page from Google?

Not reliably. It asks crawlers not to visit the page, but a blocked URL can still be known if other pages link to it.

Is robots.txt a security feature?

No. Anyone can read the file, and bad bots can ignore it. Use authentication for anything private.

What does User-agent: * mean?

It means the rules that follow apply to all crawlers.

What does an empty Disallow line mean?

It means nothing is disallowed, so crawlers may visit the whole site.

Do I need a sitemap line?

No, it is optional, but it helps crawlers discover your pages. Enter the full address, starting with https://.

Can I set different rules for different search engines?

Not with this tool, which produces one group for all crawlers. You would need to add extra groups by hand.

How often should I update it?

Review it whenever your site structure changes, such as when you add an admin area, a members section or a new store.

Does the order of the lines matter?

Within a group, the User-agent line comes first and the rules follow. The Sitemap line stands on its own and can sit at the end.

Can I use it for a site that is still in development?

Yes. Choose block everything while you build, then regenerate the file with allow everything before you launch.

Ready to Create Your File?

A correct robots.txt takes only a minute and prevents wasted crawling. Pick your default rule, list your paths, press the button and upload the result. For more helpful utilities, browse our free tools, or build your share previews with the Meta Tag Generator.