Use templates, edit user-agent groups, optionally manage AI crawlers, declare sitemaps, validate syntax, and test URLs. Everything runs in your browser. WebFixer does not receive or store your generated configuration.
Templates populate the editor with example rules. They are starting points only. CMS sites often use different paths, so review every rule before publishing.
Templates are starting points only. Edit every rule before publishing, and confirm paths match your site.
Each group lists crawlers and the paths they may or may not request.
Choose only the crawlers you want to configure. Training, search discovery and user-triggered fetches are different purposes. Blocking one crawler does not block every AI service. robots.txt is a voluntary directive, not an enforcement mechanism. Before publishing, verify each crawler token and its stated purpose against the provider's official documentation, because names and policies can change.
Optional Sitemap: lines must use absolute http(s) URLs. Build an XML sitemap with the XML Sitemap Generator if you need one.
Paste a file to parse it into the editor. Unsupported lines are kept under Additional directives. You will see a preview and must confirm before replacing your current configuration.
Browser fetch can fail when the remote site blocks cross-origin requests. If that happens, paste the file manually. This tool does not use a server-side proxy.
Preserved from import when the visual editor cannot represent a line. Verify crawler support before publishing.
This panel updates automatically as you edit. It checks syntax, sitemap URLs, file size, and risky patterns such as blocking the whole site. A green result means no problems were flagged in those checks. It does not prove every crawler will treat your file the same way, and it is not a live crawl of your website.
Live check: waiting for the first update.
Simulate whether a path would be allowed or blocked for a chosen user-agent using your current preview. This is a local rule simulator, not a live test against Google's crawlers.
The preview always matches the current configuration. Copy or download when you are ready to upload robots.txt to your site root.
robots.txt guides crawling; it does not guarantee removal from search results and is not a substitute for access control on private content.
A robots.txt file is a plain-text crawl policy published at the site root, for example https://www.example.com/robots.txt. Well-behaved crawlers request that URL and read User-agent, Allow, Disallow and optional Sitemap lines for hints about which paths they may fetch. The file is public. It is not a login wall, not encryption, and not a reliable way to remove pages from search results.
User-agent: * group.robots.txt./robots.txt.User-agent: * Disallow:
An empty Disallow usually means no path restrictions for that group. Useful on public marketing sites with nothing sensitive under crawlable URLs.
User-agent: * Disallow: /admin/
Asks matching crawlers not to fetch paths under /admin/. Linked URLs may still be discovered; use authentication for real protection.
User-agent: * Disallow: /
This asks crawlers that use the wildcard (*) group not to fetch any path. It is appropriate for many staging or private-beta setups, and the staging template warns about the pattern. It does not automatically cover every specialised crawler. Some bots, including Google's AdsBot crawlers, do not follow the * group and need their own explicit User-agent rules if you intend to restrict them. Non-compliant bots may also ignore robots.txt entirely.
User-agent: GPTBot Disallow: / User-agent: * Disallow:
Gives GPTBot a separate policy while leaving other crawlers on the wildcard group. Blocking one named bot does not block every AI service.
User-agent: * Disallow: /admin/ Allow: /admin/public/
With longest-match behaviour (as used by Google), /admin/public/ can remain allowed while other /admin/ paths stay disallowed. Use the robots.txt tester above on candidates such as /admin/, /admin/settings/ and /admin/public/.
Sitemap declarations are site-level lines. They are not part of a user-agent group and do not inherit Allow or Disallow rules from the groups above them.
Sitemap: https://www.example.com/sitemap.xml Sitemap: https://www.example.com/sitemap-news.xml
Each Sitemap: line advertises where an XML sitemap lives. The lines do not create the sitemap files.
* is the fallback group for crawlers that use it).* can match a sequence of characters in a path pattern (Google-supported).$ can anchor a pattern to the end of the path (Google-supported).Support for directives and extensions varies. Crawl-delay and Host are not treated as Google-standard rules. The robots.txt validator on this page flags many of those as warnings rather than hard errors.
Some providers publish distinct tokens for training crawls, search discovery and user-triggered fetches. Examples include GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI); ClaudeBot, Claude-SearchBot and Claude-User (Anthropic); PerplexityBot and Perplexity-User; and bingbot for Bing. Always verify the current crawler tokens and their stated purposes in each provider's official documentation before you publish rules, because names and policies can change.
Google-Extended is a standalone robots.txt control relating to Google's use of crawled content for Gemini model training and certain grounding applications. It does not control Google Search crawling or rankings. Googlebot search crawling remains separate from Google-Extended.
Use the optional AI section above to add only the agents you choose. robots.txt remains voluntary: compliant bots may honour it, but it cannot guarantee that every AI system or non-compliant bot will comply. For a separate, human-readable AI resource list, see the llms.txt Generator. llms.txt does not replace robots.txt.
Disallow: / on a public site.*.noindex when indexing is the real goal).Upload the downloaded plain-text file so it is available at https://www.example.com/robots.txt (replace with your host). Expect HTTP 200 and a text/plain (or compatible) response. Compare the live body with your generator preview. Then run WebFixer's SEO Audit on the homepage to review robots reachability, broad disallow rules, sitemap declarations and related crawl signals. You can also check DNS records with DNS Lookup if the host itself looks wrong.
The robots.txt validator highlights invalid directives, bad Sitemap URLs, duplicates, missing leading slashes, site-wide blocks and other risky patterns. The robots.txt tester applies longest-match rules with Allow winning equal-length ties, including * and trailing $ where supported. Example:
User-agent: * Disallow: /admin/ Allow: /admin/public/ Disallow: /private/
Expected local simulation: /admin/ and /admin/settings/ blocked; /admin/public/ allowed; /private/ blocked; /about/ allowed by default. This remains a browser-side simulation, not a live Googlebot lab test.
The generator does not upload files, manage CDN rules or bypass CORS when fetching remote robots.txt files.
/robots.txt.* fallback group are common.* wildcards and trailing $ end anchors in robots.txt path patterns.Run WebFixer’s free SEO audit to review robots.txt, sitemap discovery, and related crawl signals.
Copyright 2026 WebFixer.dev. All Rights Reserved