Free tools, SEO and marketing
Robots.txt generator
Choose what search engines and AI crawlers may visit, and download a ready robots.txt.
Guides
Questions
Does robots.txt hide pages from Google?
No. It tells well-behaved crawlers not to visit pages, but a blocked page can still appear in search if other sites link to it. To keep a page out of search results, allow crawling and add a noindex meta tag instead.
Should I block AI crawlers?
It is a business decision. Blocking training bots like GPTBot or ClaudeBot keeps your content out of future training data. Blocking search bots like OAI-SearchBot or PerplexityBot also means your pages won’t be cited in those AI answers, which can cost traffic.
Where does the file go?
At the root of your domain, so it opens at https://your-domain.com/robots.txt. Subdomains need their own file. Test it in Google Search Console under Settings, robots.txt.
More seo and marketing tools
- Meta tag generatorTitle, description and social tags, with Google and Facebook previews.
- Google snippet previewSee if your title and description get cut off in search results.
- Schema markup generatorJSON-LD for FAQ, articles, products, local businesses and breadcrumbs.
- .htaccess redirect generator301 redirects, HTTPS and www rules for Apache servers.
- Slug generatorClean, readable URL slugs from any title, including Cyrillic.
- Keyword density checkerWhich words and phrases a text leans on, and how often.