Any issues or updates with KCUrbEx can be found here!
Post Reply
User avatar
Posts: 35
Joined: Sat Nov 03, 2007 3:17 pm
Location: Not KC

Could someone summon youthsonic? I'd like to ask that he switch up the robots.txt to Disallow everything? I believe we simply change it to:

Code: Select all

User-agent: *
Disallow: /
User avatar
Urban Perversion
Posts: 1031
Joined: Tue Jul 05, 2005 4:58 pm
Location: KCMO

I modified the existing one, and un-commented the Disalow: * line.

It might be of some use, but not all spiders honor this, obviously. Which is why I am also adjusting the "Robots" area of the admin panel to also not allow any archiving through the form software itself
Post Reply