Options
User-agent
Sitemap
Crawl-delay
Output
Create a robots.txt file for search-engine crawlers and other automated agents. Set the user-agent, allowed and disallowed paths, crawl delay, and sitemap URL, then copy the ready-to-publish file.
Robots.txt is a crawl guidance file, not an access-control mechanism. Use authentication, permissions, or server rules when content must be private.
Place it at the root of the host, for example https://example.com/robots.txt.
No. It gives compliant crawlers instructions but does not provide security. Private content needs server-side access control.