Create robots.txt with User-agent rules, Allow/Disallow paths, and Sitemap. Validate and test crawler behavior.
See whether a path would be allowed or blocked by the generated robots.txt.
Matched rule: Disallow: /admin
User-agent: * Allow: / Disallow: /admin Disallow: /private Sitemap: https://example.com/sitemap.xml
robots.txt is a plain-text file in your site root that tells search engine crawlers which paths they may or may not request. It uses User-agent to target specific bots, Allow and Disallow for paths, and optionally Sitemap to point to your XML sitemap. It does not block access—it only advises; use server-side protection for sensitive content.
robots.txt is a file placed in your site root that tells search engine crawlers which URLs they can or cannot request. It uses directives like User-agent, Allow, and Disallow. It does not block access—it only suggests. Sensitive content should be protected by authentication, not robots.txt.
Place robots.txt at the root of your domain so it is available at https://yoursite.com/robots.txt. Search engines look for this exact URL. Our generator outputs the content; you save it as robots.txt and upload it to your server root.
The Sitemap line in robots.txt points crawlers to your XML sitemap URL. This helps search engines discover all your pages. You can list multiple Sitemap lines if you have several sitemaps. Use our XML Sitemap Generator to create the sitemap file.
Yes. Use Disallow for paths you do not want indexed (e.g. /admin, /private, /api). Never rely on robots.txt alone for security—use proper authentication and server-side access control for sensitive areas.
Yes. Add multiple User-agent blocks. Use User-agent: Googlebot for Google, User-agent: Bingbot for Bing, and User-agent: * for all other crawlers. Our tool lets you add and edit multiple user-agent blocks.