Does Silktide respect robots.txt?

What is a Robots.txt file?

A robots.txt file is a special file that tells bots what content they should avoid downloading from a website.

Note that a robots.txt file is not enforced, i.e. it is up to the bot whether it should abide by what it says.

Does Silktide respect robots.txt?

By default, Silktide purposely does not respect the robots.txt file.

This is because for most purposes we are testing whether a website functions correctly for end-users, and in our experience a large number of robots.txt files inhibit this. Many block user-facing content that would otherwise not be tested.

Note that even Google doesn’t fully abide by robots.txt, for similar reasons.

How to enforce robots.txt

If you are sure you wish to enforce your robots.txt file, you can copy the rules into the settings for your website(s). Contact us and we’ll be happy to set this up for you.

Need extra help?

Chat with our support team now and we'll be happy to help you with any issues