New: DMCA VPS in Netherland See plans
ServersCrest
Hosting
Domains
Email & Security
Servers & Licenses
Tools
More
Webmaster Tools

How to use Robots.txt Generator

1 min read1,002 views0ServersCrest, webmaster toolsپرینت

You can use this tool on the ServersCrest Tools page. Add the paths you want crawlers to allow or avoid and the public sitemap URL. Paths should match the website where you will upload the file.

Open Robots.txt Generator.

Prepare a robots.txt file

  1. Open Tools and choose Robots.txt Generator.

  2. Pick a preset or start blank.

  3. Add Disallow/Allow paths and your sitemap URL.

  4. Click Generate, then copy or download robots.txt.

Robots.txt Generator: Allow / Disallow rules; robots.txt
ServersCrest illustration using example details.

Check the rules, then save the file at that website root as robots.txt. Blocking a path is a crawler instruction, not password protection and not a guaranteed way to remove a page from search results.

How to use QR Code Generator · How to use Character & Word Counter

Need help with this? Send us a ticket and tell us which page you reached and what happened.

آیا این پاسخ به شما کمک کرد؟