Page 1 of 1

robots.txt

Posted: Thu Apr 18, 2013 11:27 am
by dannykansas
Could someone summon youthsonic? I'd like to ask that he switch up the robots.txt to Disallow everything? I believe we simply change it to:

Code: Select all

User-agent: *
Disallow: /

Re: robots.txt

Posted: Sun May 05, 2013 10:43 am
by youthsonic
I modified the existing one, and un-commented the Disalow: * line.

It might be of some use, but not all spiders honor this, obviously. Which is why I am also adjusting the "Robots" area of the admin panel to also not allow any archiving through the form software itself