Page 1 of 1
Posted: Thu Apr 18, 2013 11:27 am
Could someone summon youthsonic? I'd like to ask that he switch up the robots.txt to Disallow everything? I believe we simply change it to:
Posted: Sun May 05, 2013 10:43 am
I modified the existing one, and un-commented the Disalow: * line.
It might be of some use, but not all spiders honor this, obviously. Which is why I am also adjusting the "Robots" area of the admin panel to also not allow any archiving through the form software itself