Generate a valid robots.txt file for your website. Add user-agent rules, sitemaps, crawl delays, and use quick presets to get started fast.
OneStepToRank tracks your local search rankings across grid points, monitors competitors, and sends you alerts when positions change. See exactly where you rank on Google Maps.
Get StartedA robots.txt file is a plain text file that lives at the root of your website and tells search engine crawlers which parts of your site they can and cannot access. It follows the Robots Exclusion Protocol, an industry standard since 1994 that every major search engine -- Google, Bing, Yahoo, Yandex, and others -- respects. When a crawler arrives at your site, the first thing it does is check for yourdomain.com/robots.txt to understand your crawling preferences before visiting any other page.
The file uses simple directives to communicate with bots. User-agent specifies which crawler the rules apply to (use * for all bots). Disallow blocks specific paths from being crawled. Allow permits access to paths within a disallowed directory. Sitemap points crawlers to your XML sitemap so they can discover all your pages efficiently. Some crawlers also support Crawl-delay, which tells bots to wait a number of seconds between requests to reduce server load.
A common misconception is that robots.txt prevents pages from being indexed in search results. It does not. Disallowing a path stops crawlers from visiting that URL, but if other sites link to it, Google can still index the URL based on external signals like anchor text. To truly block a page from appearing in search results, you need a noindex meta tag or an X-Robots-Tag HTTP header. Think of robots.txt as controlling crawl access, not index visibility.
Your robots.txt file should always be placed at the root directory of your domain -- accessible at https://yourdomain.com/robots.txt. It is protocol and subdomain specific: rules for https://example.com do not apply to https://www.example.com or http://example.com. Each variant needs its own file. For most sites, including a Sitemap directive pointing to your XML sitemap is the single most valuable thing you can add, as it helps crawlers discover your content faster and more completely. Use this generator alongside our Meta Tag Generator and Schema Generator for a complete technical SEO setup.
A robots.txt file is a plain text file placed at the root of your website that tells search engine crawlers which pages or sections they are allowed or not allowed to crawl. It follows the Robots Exclusion Protocol, a standard recognized by all major search engines. The file uses directives like User-agent, Disallow, Allow, and Sitemap to control crawler behavior and manage how bots interact with your site.
No. A Disallow directive in robots.txt tells crawlers not to crawl a page, but it does not prevent that page from appearing in search results. If other websites link to a disallowed URL, Google can still index it using external information such as anchor text. To truly prevent a page from being indexed, use a noindex meta tag or X-Robots-Tag HTTP header instead. Robots.txt controls crawling access, not indexing behavior.
Your robots.txt file must be placed at the root directory of your website so it is accessible at yourdomain.com/robots.txt. The file is specific to the protocol and subdomain: https://example.com/robots.txt only controls crawling for https://example.com, not for https://www.example.com or http://example.com. If you use multiple subdomains, each one needs its own robots.txt file.
The most commonly used directives are User-agent (which crawler the rules apply to, use * for all), Disallow (blocks a path from crawling), Allow (permits crawling within a disallowed directory), Sitemap (points crawlers to your XML sitemap), and Crawl-delay (requests crawlers wait a set number of seconds between requests, supported by Bing and Yandex but ignored by Google). These directives must follow exact syntax and are case-sensitive.